More granular read-write lock requests in the run() and reproduce() methods of the Stage class #5815
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Instead of pre-emptively grabbing read locks on the dependencies and write locks on the outputs when calling the run() method, delegate the locking mechanism to other methods. This results in more selective locking that is less likely to cause a LockError when executing steps in parallel.
|
@maximerischard sorry for keeping you waiting, I assigned myself to prevent that in the future. |
efiop
approved these changes
Aug 9, 2021
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
❗ I have followed the Contributing to DVC checklist.
[]
📖 If this PR requires documentation updates, I have created a separate PR (or issue, at least) in dvc.org and linked it here.I took a stab at making the read-write lock requests in the run() and reproduce() methods of the Stage class more granular, in order to address issues #5795 and #5007. This makes the “manually parallel” execution of dvc steps possible.