project: Support .gitignore to exclude directories while searching for stage files

[ghost]

This prevents walking directories like node_modules.

Maybe this is not the way to speed up the stage-file search but it may help.
It could be more harmful than helpful, so I open this issue for discussion.

[ghost]

Another strat would be to store every stage created in the state, but DVC would need a command to refresh or sync the state file with the current workspace (maybe is not worth trading the ease of usage for some speed)

[efiop]

Using state is a great idea. Introducing refresh or sync commands doesn't seem right though :). If we were to go with this strat, we could just save mtimes for directories and rescan them only if they've changed. This is the same as we track dep/out directories right now.

That being said, using .gitignore seems like a very reasonable, simple and elegant approach. Especially since it also covers directories that change frequently and won't require us to rescan it every time, like the state approach would.

[shcheklein]

I like the idea with .gitignore!

I think we can parallelize the search (if it's not yet, of course :)) and cache mtimes indeed, that alone should make some difference.

[efiop]

Related to #1820 . Adding node_modules to .dvcignore would prevent dvc from searching for dvcfiles there.

It seems unlikely, but I could imagine someone adding dvcfiles to gitignore and yet want dvc to discover them. Using .dvcignore seems like a more direct and clear approach.

[ghost]

Indeed, @efiop , but it is like an extra step that the user needs to do.

There are a lot of people adding their virtual environments to docker images because they forgot or might not know about .dockerignore. Is not like a huge deal, in this case, searching through the virtual environment, node modules, or even the .git directory would add max a few seconds, maybe something unnoticeable.

[ghost]

Maybe we can include it in an "Advanced Users" guide, with information about reflinks, and more performance enhancements.

[efiop]

Indeed, @efiop , but it is like an extra step that the user needs to do.

@MrOutis True. If I remember correctly, in our .dvcignore implementation we have !path supported, which would allow to explicitly stop ignoring something. If we decide to go with the approach that you suggest, we could make that !path in dvcignore make dvc stop ignoring particular path even if it is in .gitignore. I'm just not sure if that approach is ok to make default. Need to give it a better thought...

[ghost]

Need to give it a better thought...

Agree, @efiop , let's leave this open and see how it goes with .dvcignore

[efiop]

Seems like .dvcignore does the job. Let's close this for now, I don't see any users requesting this specifically. If anyone will request it - please feel free to reopen.