Skip to content

Clean up buffer inside DataPipe instance during destruction #370

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
ejguan opened this issue Apr 25, 2022 · 3 comments
Closed

Clean up buffer inside DataPipe instance during destruction #370

ejguan opened this issue Apr 25, 2022 · 3 comments

Comments

@ejguan
Copy link
Contributor

ejguan commented Apr 25, 2022

🚀 The feature

Could we add __del__ to clean up buffer, which are attached to DataPipe instance.

  • Demux
  • Fork
  • etc

Motivation, pitch

We may encounter the problem that the objects inside buffer are not clean up by DataPipe instance itself but relied on gc to clean up.
This could potentially to resolve the problem from pytorch/vision#5801 (comment)

TorchVision chose to ignore the ResourceWarning for the test suite. But, we still need to consider for other domains or users who want to keep these warnings.

Alternatives

No response

Additional context

No response

cc: @NivekT

@NivekT
Copy link
Contributor

NivekT commented Apr 25, 2022

I can submit a PR for this. I gave this some thoughts, I think calling clear() should clear the buffer without deleting the underlying objects so that if there are other references to those objects, they will still work properly.

@ejguan
Copy link
Contributor Author

ejguan commented Apr 25, 2022

That makes sense to me.

@NivekT
Copy link
Contributor

NivekT commented Apr 26, 2022

Fixed by pytorch/pytorch#76345

@NivekT NivekT closed this as completed Apr 26, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants