Skip to content

Auto dependency calculation --auto-dag #130

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
dmpetrov opened this issue Jul 31, 2017 · 4 comments
Closed

Auto dependency calculation --auto-dag #130

dmpetrov opened this issue Jul 31, 2017 · 4 comments
Assignees
Labels
question I have a question?

Comments

@dmpetrov
Copy link
Member

Currently DVC implicitly derives dependencies based on command line args.
$ dvc run python mycode.py data/input.csv data/output.p
It goes like this:

  1. If arg (let say data/output.p) looks like data file (in data dir) we assume it is an input parameter.
  2. If a data file was changed we change our mind and say the arg is an output.

Also, we have a special args like --input or -i and --output or -o for explicit input\output definition.

Should we keep auto\implicit dependency calculation or we should hide this logic behind, let say --auto-dag or -a option?

Result will be:
$ dvc run --auto-dag python mycode.py data/input.csv data/output.p
or explicitly:
$ dvc run -i data/input.csv -o data/output.p python mycode.py data/input.csv data/output.p

@dmpetrov dmpetrov added the question I have a question? label Jul 31, 2017
@dmpetrov dmpetrov added this to the dvc July-17 milestone Jul 31, 2017
@dmpetrov
Copy link
Member Author

Actually, we use two step dependency calculation:

  1. Command line based.
  2. Repository state based (before and after command run).

Both of them make DVC logic quite fragile.

To make DVC more persistent we should probably turn off both of the options by default. New parameters --auto-args, --auto-git or combined --auto-dag will include one of these.

And best practice should be - specify your inputs and outputs explicitly.

@dmpetrov
Copy link
Member Author

One more specific case #154

@efiop efiop removed this from the dvc August-28 milestone Aug 27, 2017
@efiop efiop self-assigned this Aug 27, 2017
@efiop
Copy link
Contributor

efiop commented Aug 27, 2017

Add --no-auto-dag as a temporary solution.

@efiop
Copy link
Contributor

efiop commented Jan 4, 2018

No longer relevant, as we switched to explicit dependencies.

@efiop efiop closed this as completed Jan 4, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
question I have a question?
Projects
None yet
Development

No branches or pull requests

2 participants