-
Notifications
You must be signed in to change notification settings - Fork 1.2k
Brancher refactor #1709
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Brancher refactor #1709
Changes from all commits
Commits
Show all changes
8 commits
Select commit
Hold shift + click to select a range
e782344
Move dvc/scm/git.py to dvc/scm/git/__init__.py
ei-grad 9e6c520
Abstract access to file objects
ei-grad 8c8b585
tests: test_install commit and clean
ei-grad 16f74f6
tests: test_metrics missing .dvc files in commit
ei-grad 8fa968c
collect: nx.dfs_postorder_nodes yeilds the passed node
ei-grad 2ef3a7c
tests: windows has different linesep
ei-grad 63dd3f2
tests: fix get_gitignore_content
ei-grad 9219398
tests: fix #1754
ei-grad File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,56 @@ | ||
def brancher( # noqa: E302 | ||
self, branches=None, all_branches=False, tags=None, all_tags=False | ||
): | ||
"""Generator that iterates over specified revisions. | ||
|
||
Args: | ||
branches (list): a list of branches to iterate over. | ||
all_branches (bool): iterate over all available branches. | ||
tags (list): a list of tags to iterate over. | ||
all_tags (bool): iterate over all available tags. | ||
|
||
Yields: | ||
str: the display name for the currently selected tree, it could be: | ||
- a git revision identifier | ||
- empty string it there is no branches to iterate over | ||
- "Working Tree" if there are uncommited changes in the SCM repo | ||
""" | ||
if not any([branches, all_branches, tags, all_tags]): | ||
yield "" | ||
return | ||
|
||
saved_tree = self.tree | ||
revs = [] | ||
|
||
scm = self.scm | ||
|
||
if self.scm.is_dirty(): | ||
from dvc.repo.tree import WorkingTree | ||
|
||
self.tree = WorkingTree() | ||
yield "Working Tree" | ||
|
||
if all_branches: | ||
branches = scm.list_branches() | ||
|
||
if all_tags: | ||
tags = scm.list_tags() | ||
|
||
if branches is None: | ||
revs.extend([scm.active_branch()]) | ||
else: | ||
revs.extend(branches) | ||
|
||
if tags is not None: | ||
revs.extend(tags) | ||
|
||
# NOTE: it might be a good idea to wrap this loop in try/finally block | ||
# to don't leave the tree on some unexpected branch after the | ||
# `brancher()`, but this could cause problems on exception handling | ||
# code which might expect the tree on which exception was raised to | ||
# stay in place. This behavior is a subject to change. | ||
for rev in revs: | ||
self.tree = scm.get_tree(rev) | ||
yield rev | ||
|
||
self.tree = saved_tree |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,65 @@ | ||
import os | ||
|
||
from dvc.utils.compat import open | ||
|
||
|
||
class BaseTree(object): | ||
"""Abstract class to represent access to files""" | ||
|
||
def open(self, path): | ||
"""Open file and return a stream.""" | ||
|
||
def exists(self, path): | ||
"""Test whether a path exists.""" | ||
|
||
def isdir(self, path): | ||
"""Return true if the pathname refers to an existing directory.""" | ||
|
||
def isfile(self, path): | ||
"""Test whether a path is a regular file""" | ||
|
||
def walk(self, top, topdown=True): | ||
"""Directory tree generator. | ||
|
||
See `os.walk` for the docs. Differences: | ||
- no support for symlinks | ||
- it could raise exceptions, there is no onerror argument | ||
""" | ||
|
||
|
||
class WorkingTree(BaseTree): | ||
"""Proxies the repo file access methods to working tree files""" | ||
|
||
def open(self, path): | ||
"""Open file and return a stream.""" | ||
return open(path, encoding="utf-8") | ||
|
||
def exists(self, path): | ||
"""Test whether a path exists.""" | ||
return os.path.exists(path) | ||
|
||
def isdir(self, path): | ||
"""Return true if the pathname refers to an existing directory.""" | ||
return os.path.isdir(path) | ||
|
||
def isfile(self, path): | ||
"""Test whether a path is a regular file""" | ||
return os.path.isfile(path) | ||
|
||
def walk(self, top, topdown=True): | ||
"""Directory tree generator. | ||
|
||
See `os.walk` for the docs. Differences: | ||
- no support for symlinks | ||
- it could raise exceptions, there is no onerror argument | ||
""" | ||
|
||
def onerror(e): | ||
raise e | ||
|
||
for root, dirs, files in os.walk( | ||
top, topdown=topdown, onerror=onerror | ||
): | ||
if topdown: | ||
dirs[:] = [i for i in dirs if i not in (".git", ".hg", ".dvc")] | ||
yield os.path.normpath(root), dirs, files |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This breaks tests that expect previous behaviour of yielding "" if it is not from any branch. Could we continue doing that?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oops. Not this one actually, but the same thing in line 16. I did run the tests many times, but not after the line 16 change which I added at the last moment :-/.
I think it is better to clarify in the docstring that what the brancher yeilds is not a branch names (instead it is just the abstract name of the currently selected tree), and adjust the tests for this behavior. What do you think? @efiop @shcheklein
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
AFAIR it is currently used only to display the name for metrics in different branches in
dvc metrics show -a
output.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oh... and in
dvc push
output, where it is reduntant for sure...Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So I think it is still ok to leave the "Working Tree" string for iteration with
--all-branches
, but yield an empty string when there is no things to iterate over.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@ei-grad "index" is even more confusing IMHO :) Empty string feels nice, but maybe I'm just too used to it 🙂
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Updated the docstring to about what the brancher could yeild.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
so, this is about printing a tree name when we do some output, right? First, I would make a tree itself return the name, no need then to yield anything and pass it around, we will be able to access it from the repo, the same way we access the current object. Is it possible? Second, in the FsTree we can return "Working Tree" or "*" or "workspace", or whatever :) Btw, i'm more or less fine with the current solution as well, it's not ideal but it's not hard to fix anyway. As for the empty string. @efiop does it mean we will have to do something like
if not branch: name "Working Tree"
further along, in multiple place? I would try to avoid that.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@shcheklein No, I don't see why we even need "Working Tree" anywhere.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In
dvc metrics show -a
when we print output we need a way to distinguish "workspace" from an actual branch. May be some other places, @ei-grad should know better than me by now :)