LocalRemoteTree: use repo tree as work_tree with local outputs #4125

pared · 2020-06-26T21:18:06Z

❗ I have followed the Contributing to DVC checklist.
📖 If this PR requires documentation updates, I have created a separate PR (or issue, at least) in dvc.org and linked it here.
❌ I will check DeepSource, CodeClimate, and other sanity checks below. (We consider them recommendatory and don't expect everything to be addressed. Please fix things that actually improve code or fix bugs.)

Thank you for the contribution - we'll try to review it as soon as possible. 🙏

EDIT:
Fixes #4110
Fixes #4197

pared · 2020-06-26T21:20:29Z

tests/func/test_ignore.py

@@ -173,3 +173,27 @@ def test_ignore_blank_line(tmp_dir, dvc):
    tmp_dir.gen(DvcIgnore.DVCIGNORE_FILE, "foo\n\ndir/ignored")

    assert _files_set("dir", dvc.tree) == {"dir/other"}
+
+
+def test_ignore_in_added_dir(tmp_dir, dvc):


I did consider making test for LocalRemoteTree only, but I guess, in the end, we want to be sure that one can actually ignore something in added directory.

pmrowla

I think the real underlying issue in #4110 is that we still use "remotes" in DVC outputs rather than purely using trees.

This causes problems for LocalOutput, since we treat regular local DVC output paths the same way as a local external dependency (we handle them both using LocalRemoteTree) even though regular output paths should be repo tree paths rather than remote paths.

So for a regular dvc added dir output, right now when we run

    def save_info(self):
        return self.remote.save_info(self.path_info)

we eventually walk the LocalRemoteTree to determine what files should go in dir cache and generate our dir hash. This is what causes the bug w/not using DVC ignore for that directory, and why wrapping the remote work tree w/cleantree fixes the bug.

I think what we should really be doing is treating regular outputs separately from local external dependencies.

We do want to use CleanTree for regular outputs, since we are dealing with actual DVC repo paths (I'm not sure whether that means we use a RepoTree or wrapping LocalRemoteTree work tree w/CleanTree for regular outputs).

But for other types of local "remote" paths (including external dependencies) we should not need to use CleanTree.

pmrowla · 2020-06-27T02:26:56Z

dvc/ignore.py

@@ -216,7 +216,7 @@ def _parents_exist(self, path):
                [os.path.abspath(path), self.tree_root]
            )
        ):
-            return False
+            return True


This works as a temporary solution, but it still seems unintuitive.

It seems to me that dvcignore should only ever apply to a DVC repo, and by definition anything outside the repo root directory should not exist in the clean tree.

pmrowla · 2020-06-27T02:35:08Z

dvc/remote/local.py

@@ -66,7 +67,7 @@ def work_tree(self):
        # GitTree arbitarily. When repo.tree is GitTree, local cache needs to
        # use its own WorkingTree instance.
        if self.repo:
-            return WorkingTree(self.repo.root_dir)
+            return CleanTree(WorkingTree(self.repo.root_dir))


Same as before, this works as part of a temporary fix but I'm not sure it makes sense as a real solution.

For local remotes, and DVC local cache we should need to wrap the work tree with CleanTree. With a local remote, it should only contain files which have been pushed to it, and we should not care about filtering/ignoring anything inside the remote. Likewise, for cache, we should not care about filtering anything inside .dvc/cache.

The main point of this tree refactor was that local remotes and local cache should only be dealing with paths inside the actual remote or cache. We run into issues when we start mixing remote/cache/repo tree paths and treating them all as just "local filesystem paths".

pmrowla · 2020-06-27T02:55:39Z

These changes do fix the user issue, and CleanTree used to allow paths outside the repo root before the remote tree refactoring, so I'd be ok with merging this as a bug fix for now. But long term I think we should keep the other things I mentioned in mind.

pared · 2020-07-14T10:46:09Z

@pmrowla I tried to make it less hackish behavior, and determine what tree should be used on Output level. The results are in newest commit.

I do believe that current behavior is still hackish.
What I believe we should do after this change:

rename *RemoteTree or (Git/Working/Repo/DVC/Clean)Tree - it gets really confusing which one we are using and where, and we are intermixing them, especially in LocalRemoteTree. Not sure how the renaming should go, so far I came up with (Git/Working/Repo/DVC)FileSystem since those objects provide fs-like operations, thought in that case CleanTree does not make sense, maybe it should be named IgnoreWrapper or something like that.
Related to 1: we are duplicating Local Filesystem functionality in WorkingTree and LocalRemoteTree. I think we should get rid of that duplication. First thing that comes to my mind is creating more BaseTree children, implementing fs operations for all remote types, and using those trees in *RemoteTree's. That does not seem like perfect soultion as we already have this chierarchy *Remote -> *RemoteTree and now we would add another thing: *RemoteTree -> *Tree. Probably that should be reconsidered.

efiop · 2020-07-14T12:40:25Z

dvc/output/local.py

+    @cached_property
+    def remote(self):
+        if self._remote:
+            return self._remote
+
+        work_tree = None
+        if self.repo and self.is_in_repo and is_working_tree(self.repo.tree):
+            work_tree = self.repo.tree
+        tree = LocalRemoteTree(self.repo, {}, work_tree)
+        return LocalRemote(tree)


Could we do this remote setting in __init__? Just feels a bit weird introducing cached property & _remote mix in both here and base class for this hack.

We could, but in that case, we will set remote inside parent constructor and override it here. Thats what I wanted to avoid creating cached_property

@pared but we can set it after super(), no? Or pass the new instance to the constructor.

efiop · 2020-07-14T12:50:32Z

Had a similar issue with config and it really hurts 🙁 Thinking that maybe we should tackle #4050 right away and get rid of that is_working_tree mess once and for all. Otherwise, we might be introducing much more hidden issues than we are solving (though in this case this is clearly very serious).

Let's discuss this during planning today, maybe someone will be able to look into this part of trees deeply ASAP. If not, we'll have to merge the hack, indeed. 🙁

tests/func/test_ignore.py

pared commented Jun 26, 2020

View reviewed changes

efiop requested a review from pmrowla June 26, 2020 21:40

pared changed the title ~~4110 dvcignore issue~~ [WIP] LocalRemoteTree: use CleanTree with work_tree Jun 26, 2020

pmrowla reviewed Jun 27, 2020

View reviewed changes

pared force-pushed the 4110_dvcignore_issue branch from e311fa1 to 8000d80 Compare July 8, 2020 11:02

weekly-digest bot mentioned this pull request Jul 12, 2020

Weekly Digest (5 July, 2020 - 12 July, 2020) #4195

Closed

pared mentioned this pull request Jul 13, 2020

dvc add: .dvcignore example not working as expected [qa] #4197

Closed

pared force-pushed the 4110_dvcignore_issue branch from 8000d80 to ce61e28 Compare July 14, 2020 10:30

pared marked this pull request as ready for review July 14, 2020 10:31

pared changed the title ~~[WIP] LocalRemoteTree: use CleanTree with work_tree~~ LocalRemoteTree: use repo tree as work_tree with local outputs Jul 14, 2020

pared requested review from efiop and skshetry July 14, 2020 10:46

efiop requested a review from pmrowla July 14, 2020 12:30

efiop reviewed Jul 14, 2020

View reviewed changes

pared force-pushed the 4110_dvcignore_issue branch from ce61e28 to aaea4e2 Compare July 15, 2020 08:22

pared requested a review from efiop July 15, 2020 08:23

efiop reviewed Jul 16, 2020

View reviewed changes

tests/func/test_ignore.py Outdated Show resolved Hide resolved

output: local: use repo tree if possible

99ad242

pared force-pushed the 4110_dvcignore_issue branch 2 times, most recently from aaea4e2 to 99ad242 Compare July 16, 2020 13:04

pared requested a review from efiop July 16, 2020 13:42

pmrowla approved these changes Jul 16, 2020

View reviewed changes

efiop approved these changes Jul 16, 2020

View reviewed changes

efiop merged commit 91f3036 into iterative:master Jul 16, 2020

weekly-digest bot mentioned this pull request Jul 19, 2020

Weekly Digest (12 July, 2020 - 19 July, 2020) #4238

Closed

pared mentioned this pull request Jul 21, 2020

dvc 1.x tries to remove files in .dvcignore #4249

Closed

pared deleted the 4110_dvcignore_issue branch January 4, 2022 10:13

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

LocalRemoteTree: use repo tree as work_tree with local outputs #4125

LocalRemoteTree: use repo tree as work_tree with local outputs #4125

Uh oh!

pared commented Jun 26, 2020 •

edited

Loading

Uh oh!

pared Jun 26, 2020

Uh oh!

pmrowla left a comment

Uh oh!

pmrowla Jun 27, 2020

Uh oh!

pmrowla Jun 27, 2020

Uh oh!

pmrowla commented Jun 27, 2020

Uh oh!

pared commented Jul 14, 2020

Uh oh!

efiop Jul 14, 2020

Uh oh!

pared Jul 14, 2020

Uh oh!

efiop Jul 14, 2020 •

edited

Loading

Uh oh!

efiop commented Jul 14, 2020

Uh oh!

Uh oh!

Uh oh!

LocalRemoteTree: use repo tree as work_tree with local outputs #4125

LocalRemoteTree: use repo tree as work_tree with local outputs #4125

Uh oh!

Conversation

pared commented Jun 26, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pared Jun 26, 2020

Choose a reason for hiding this comment

Uh oh!

pmrowla left a comment

Choose a reason for hiding this comment

Uh oh!

pmrowla Jun 27, 2020

Choose a reason for hiding this comment

Uh oh!

pmrowla Jun 27, 2020

Choose a reason for hiding this comment

Uh oh!

pmrowla commented Jun 27, 2020

Uh oh!

pared commented Jul 14, 2020

Uh oh!

efiop Jul 14, 2020

Choose a reason for hiding this comment

Uh oh!

pared Jul 14, 2020

Choose a reason for hiding this comment

Uh oh!

efiop Jul 14, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

efiop commented Jul 14, 2020

Uh oh!

Uh oh!

Uh oh!

pared commented Jun 26, 2020 •

edited

Loading

efiop Jul 14, 2020 •

edited

Loading