cmd: --to-cache/remote updates (add, import-url, update)

[isidentical]

• add docs for dvc {add, import-url} --to-remote (Add docs regaring --to-remote option for add/import-url #2091)
• add docs for straight-to-cache (including --out update ref: add --out for local targets #2220) (cmd: add to-cache docs #2246)
• add docs for dvc update --to-remote (cmd: add dvc update --to-remote docs #2175)
  • mention update --to-remote doesn't work with repo imports (only from other external sources i.e. import-urled) (cmd: explicitly mention update is forbidden for repo imports #2237)
• mention about --to-remote in https://dvc.org/doc/user-guide/managing-external-data
• cmd: update add --to-remote similar to import-url and... #2161 (Example for dvc add --to-remote #2172)
• Differentiate https://dvc.org/doc/command-reference/add#example-transfer-to-remote-storage from https://dvc.org/doc/command-reference/import-url#example-transfer-to-remote-storage per Example for dvc add --to-remote #2172 (comment):

  By default when you do import-url DVC downloads data into the workspace so that it can be saved into cache, and later into the remote storage. That's important to preserve it since we want to keep the project reproducible. In some situations though you might not have enough space on the machine you are running import-url, but you still want this data to be saved into remote storage, you still want this data be accessible through regular commands like dvc pull (e.g. to run the pipeline on another machine that has enough space in cache, or when a large shared cache is being used, etc). In those cases, to "bootstrap" the project it's handy to use --to-remote ....

[jorgeorpinel]

mention about --to-remote in https://dvc.org/doc/user-guide/managing-external-data

No need as we have a big ticket to rewrite that whole thing. But I did add another checkbox per #2175 (review) 👍

[dberenbaum]

mention about --to-remote in https://dvc.org/doc/user-guide/managing-external-data

No need as we have a big ticket to rewrite that whole thing. But I did add another checkbox per #2175 (review) 👍

So what is the plan with https://dvc.org/doc/user-guide/managing-external-data for now? I think we have agreed that it's not worth re-writing for 2.0, since we need to change the functionality anyway, probably for 3.0. So that leaves us with some suboptimal choices:

• Keep it as is
• Drop it completely
• Find ways to further discourage users from relying on it (remove or replace links to it, find a way to make the warning at the top even more prominent, etc.)

[jorgeorpinel]

Unassigned @isidentical since the only thing left is to differentiate the to-remote exampled in add vs import-url, which we should be able to do (that said feel free to contribute a draft @isidentical).

what is the plan with https://dvc.org/doc/user-guide/managing-external-data

@dberenbaum please check/repost in #520 🙂

[jorgeorpinel]

Removed epic label since only one task is pending. WDYT about the remaining check box @dberenbaum ? Thanks

[dberenbaum]

Are add --to-remote and import-url --to-remote that different? It seems the main difference is whether the user needs to track updates to that source location, but that's more about import-url generally than anything specific to --to-remote. I don't see the need to prioritize this checkbox unless it's about explaining --to-remote differently in both add and import-url.

Maybe @shcheklein has some idea since I think this was his comment?

[shcheklein]

To be honest it's hard to restore the whole context now :) (partially that's why it's better to address certain issues earlier if possible - after that it takes time from a lot of people to read it again and restore the context). It's fine to skip it now I think. Probably I had in mind some general improvements to both intros to make them more explicit and simpler to read. I don't my suggestion though at the moment- it needs more time.

Jorge, you had initially concern that those two are completely different. To some extent they are, but intros phrases correct at the moment - import vs add, etc. So they are already different if you familiar with DVC terminology - what import does, what add does, etc.

WDYT?

[jorgeorpinel]

the main difference is whether the user needs to track updates to that source location

This is the only difference technically, yes (extra info. in the "import .dvc file").

you had initially concern that those two are completely different

Initially I argued that we only need the option in import-url but keeping it in add too was decided because "they're different operations", so I was trying to reflect that in docs.

better to address certain issues earlier if possible

Sometimes letting time pass helps get perspective though. I think we were not understanding each other well before. I see value in having the option in both places now, even if it's just for discoverability (since add is much more used than import). I guess that was your argument and no that they're different use cases (what I understood).

So I'll close this and just review that the language is consistent and that each command semantics are preserved in both examples. Another strategy could be to only keep the example in import-url and link to it from the add --to-remote option. Let's decide on the PR (I'll tag you).

Thanks