cloud versioning: external outputs

[dberenbaum]

For a cloud bucket/container with versioning turned on, external outputs can work like cloud-versioned remotes to simplify the workflow. No external cache is needed since a cloud-versioned external output path can serve as its own remote/cache. DVC should only need to collect the version info to store in .dvc or dvc.lock. This also can make external outputs much more performant because no copies or md5s are needed.

Question: Should dvc checkout update the current versions of external outputs? This should probably be configurable, but it can be left for a follow-up task.

Out of scope: Isolation of external output paths from race conditions and conflicts when multiple users are writing to the same paths. Let's improve the current functionality before worrying about this, which we could address either with something like #3920 or adding locks on the cloud.

[dberenbaum]

To clarify, this is mostly needed in dvc.lock for cloud-focused pipelines. For data management without pipelines, import-url [--now-download] can already be used for most scenarios to add data from cloud.

[dberenbaum]

@pmrowla Let's leave this out of scope for now. I think we can come back to it after we release the initial cloud versioning features.

[omesser]

Hey @dberenbaum - for Q1 2023 - my understanding, scope is as follows:

must have

• collect the version info of external outputs to store in .dvc or dvc.lock

nice to have / followups

• dvc checkout to update versions of external outputs - configurable, ( non default?)
• dealing with multi-user race conditions

Correct me if I got it wrong or some things missing / changed / need clarification

[dberenbaum]

Some questions/thoughts:

• Is any flag needed in dvc stage add to use cloud versioning for an output, or should this be handled automatically?
• Should it be possible to download/pull the output?
• What should checkout do?
• What fields (if any) should be added to the dvc.yaml for a cloud-versioned output?