[CUDACachingAlloc/GPUInference] Implement garbage collection without GPU sync #74261
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
CI Flow Status⚛️ CI FlowRuleset - Version:
|
🔗 Helpful links
💊 CI failures summary and remediationsAs of commit a94589f (more details on the Dr. CI page): 💚 💚 Looks good so far! There are no failures yet. 💚 💚 This comment was automatically generated by Dr. CI (expand for details).Please report bugs/suggestions to the (internal) Dr. CI Users group. |
|
This pull request was exported from Phabricator. Differential Revision: D34482514 |
|
cc @mcarilli |
|
This pull request was exported from Phabricator. Differential Revision: D34482514 |
jaewonlee-fb
force-pushed
the
export-D34482514
branch
from
87b11d7 to
4259d76
Compare
|
This pull request was exported from Phabricator. Differential Revision: D34482514 |
jaewonlee-fb
force-pushed
the
export-D34482514
branch
from
4259d76 to
e8666d3
Compare
|
This pull request was exported from Phabricator. Differential Revision: D34482514 |
jaewonlee-fb
force-pushed
the
export-D34482514
branch
from
e8666d3 to
915860d
Compare
|
This pull request was exported from Phabricator. Differential Revision: D34482514 |
jaewonlee-fb
force-pushed
the
export-D34482514
branch
from
915860d to
e567d2c
Compare
|
This pull request was exported from Phabricator. Differential Revision: D34482514 |
jaewonlee-fb
force-pushed
the
export-D34482514
branch
from
e567d2c to
1624443
Compare
|
This pull request was exported from Phabricator. Differential Revision: D34482514 |
jaewonlee-fb
force-pushed
the
export-D34482514
branch
from
1624443 to
14c5db7
Compare
|
This pull request was exported from Phabricator. Differential Revision: D34482514 |
jaewonlee-fb
force-pushed
the
export-D34482514
branch
from
14c5db7 to
f0a756f
Compare
|
This pull request was exported from Phabricator. Differential Revision: D34482514 |
jaewonlee-fb
force-pushed
the
export-D34482514
branch
from
f0a756f to
803e2c2
Compare
|
This pull request was exported from Phabricator. Differential Revision: D34482514 |
jaewonlee-fb
force-pushed
the
export-D34482514
branch
from
803e2c2 to
9f82fd2
Compare
jaewonlee-fb
force-pushed
the
export-D34482514
branch
from
9f82fd2 to
3b86ea0
Compare
|
This pull request was exported from Phabricator. Differential Revision: D34482514 |
|
cc @mwootton |
…GPU sync (pytorch#74261) Summary: Pull Request resolved: pytorch#74261 ### Goal Implement a cheap way to reclaim GPU memory (garbage collection) without incurring GPU sync. ### Why do we need this? Currently, there are only two ways to reclaim GPU memory block already assigned to a particular stream. - `release_available_cached_blocks(params)`: Free blocks exceeding the `CachingAllocatorConfig::max_split_size()` until we can satisfy the request. Issue: If the `max_split_size` is unset (default), this function is a no-op. Even if this is set, the reclamation is quite conservative (e.g., never frees blocks under max_split_size). - `release_cached_blocks()`: Waits for all the in-flight events and then reclaim blocks. Issue: 'waiting for all event' is very expensive as it will likely stall all the GPU operations. Many GPU applications without a proper handling of potential GPU throttling would suffer/crash. ### Proposed idea - If the garbage collection threshold is set, try to reclaim some memory blocks *without* synchronization. It should be safe to do so, as `release_available_cached_blocks` essentially does the same thing (but less aggressively). - GC is triggered only when we fail to serve a `malloc` request from the block pool. No need to free blocks when the block pool is functioning just fine. - Prioritize reclaiming blocks that weren't reused for long time. Reclamation stops once the used memory capacity < threshold. - This code path is totally optional; by default it won't be invoked. Test Plan: - Unit tests - Manually checked that the GPU memory usage stays as indicated by the garbage collector. If not the caching allocator at least tries to keep freeing the blocks. Reviewed By: jianyuh Differential Revision: D34482514 fbshipit-source-id: 1d29b589752d489ecbf9bd7d27fb624f980e5666
|
This pull request was exported from Phabricator. Differential Revision: D34482514 |
jaewonlee-fb
force-pushed
the
export-D34482514
branch
from
3b86ea0 to
a94589f
Compare
facebook-github-bot
pushed a commit
that referenced
this pull request
Mar 21, 2022
…GPU sync (#74261) Summary: Pull Request resolved: #74261 ### Goal Implement a cheap way to reclaim GPU memory (garbage collection) without incurring GPU sync. ### Why do we need this? Currently, there are only two ways to reclaim GPU memory block already assigned to a particular stream. - `release_available_cached_blocks(params)`: Free blocks exceeding the `CachingAllocatorConfig::max_split_size()` until we can satisfy the request. Issue: If the `max_split_size` is unset (default), this function is a no-op. Even if this is set, the reclamation is quite conservative (e.g., never frees blocks under max_split_size). - `release_cached_blocks()`: Waits for all the in-flight events and then reclaim blocks. Issue: 'waiting for all event' is very expensive as it will likely stall all the GPU operations. Many GPU applications without a proper handling of potential GPU throttling would suffer/crash. ### Proposed idea - If the garbage collection threshold is set, try to reclaim some memory blocks *without* synchronization. It should be safe to do so, as `release_available_cached_blocks` essentially does the same thing (but less aggressively). - GC is triggered only when we fail to serve a `malloc` request from the block pool. No need to free blocks when the block pool is functioning just fine. - Prioritize reclaiming blocks that weren't reused for long time. Reclamation stops once the used memory capacity < threshold. - This code path is totally optional; by default it won't be invoked. Test Plan: - Unit tests - Manually checked that the GPU memory usage stays as indicated by the garbage collector. If not the caching allocator at least tries to keep freeing the blocks. Reviewed By: jianyuh Differential Revision: D34482514 fbshipit-source-id: d5eae62ac60b94b0bca851f9d233a092d086e3c2
|
Hey @jaewonlee-fb. |
shahofblah
pushed a commit
that referenced
this pull request
Mar 25, 2022
…GPU sync (#74261) Summary: Pull Request resolved: #74261 ### Goal Implement a cheap way to reclaim GPU memory (garbage collection) without incurring GPU sync. ### Why do we need this? Currently, there are only two ways to reclaim GPU memory block already assigned to a particular stream. - `release_available_cached_blocks(params)`: Free blocks exceeding the `CachingAllocatorConfig::max_split_size()` until we can satisfy the request. Issue: If the `max_split_size` is unset (default), this function is a no-op. Even if this is set, the reclamation is quite conservative (e.g., never frees blocks under max_split_size). - `release_cached_blocks()`: Waits for all the in-flight events and then reclaim blocks. Issue: 'waiting for all event' is very expensive as it will likely stall all the GPU operations. Many GPU applications without a proper handling of potential GPU throttling would suffer/crash. ### Proposed idea - If the garbage collection threshold is set, try to reclaim some memory blocks *without* synchronization. It should be safe to do so, as `release_available_cached_blocks` essentially does the same thing (but less aggressively). - GC is triggered only when we fail to serve a `malloc` request from the block pool. No need to free blocks when the block pool is functioning just fine. - Prioritize reclaiming blocks that weren't reused for long time. Reclamation stops once the used memory capacity < threshold. - This code path is totally optional; by default it won't be invoked. Test Plan: - Unit tests - Manually checked that the GPU memory usage stays as indicated by the garbage collector. If not the caching allocator at least tries to keep freeing the blocks. Reviewed By: jianyuh Differential Revision: D34482514 fbshipit-source-id: d5eae62ac60b94b0bca851f9d233a092d086e3c2 (cherry picked from commit 05780f1)
Jack47
reviewed
Jul 24, 2022
| @@ -102,6 +102,7 @@ struct DeviceStats { | |||
| // cudaMalloc).. | |||
| struct BlockInfo { | |||
| int64_t size = 0; | |||
| int32_t gc_counter = 0; | |||
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary:
Goal
Implement a cheap way to reclaim GPU memory (garbage collection) without incurring GPU sync.
Why do we need this?
Currently, there are only two ways to reclaim GPU memory block already assigned to a particular stream.
release_available_cached_blocks(params): Free blocks exceeding theCachingAllocatorConfig::max_split_size()until we can satisfy the request.Issue: If the
max_split_sizeis unset (default), this function is a no-op. Even if this is set, the reclamation is quite conservative (e.g., never frees blocks under max_split_size).release_cached_blocks(): Waits for all the in-flight events and then reclaim blocks.Issue: 'waiting for all event' is very expensive as it will likely stall all the GPU operations. Many GPU applications without a proper handling of potential GPU throttling would suffer/crash.
Proposed idea
release_available_cached_blocksessentially does the same thing (but less aggressively).mallocrequest from the block pool. No need to free blocks when the block pool is functioning just fine.Test Plan:
Differential Revision: D34482514