cloud versioning: multiple remotes/metadata format

[dberenbaum]

Part of #7995

Description

Cloud versioning does not support multiple remotes.

Reproduction

You can reproduce with the script below. Replace CLOUD_REMOTE_1 and CLOUD_REMOTE_2 with your own s3 paths and configure your credentials.

BUCKET=mybucket
REMOTE_1=remote1
REMOTE_2=remote2

export AWS_PROFILE=iterative-sandbox

echo "Get repo."
rm -rf repo
git init repo
cd repo
dvc init

echo "Add two cloud-versioned DVC remotes."
dvc remote add -d cloud-1 s3://$BUCKET/$REMOTE_1
dvc remote modify cloud-1 version_aware true
dvc remote modify cloud-1 worktree true
dvc remote add cloud-2 s3://$BUCKET/$REMOTE_2
dvc remote modify cloud-2 version_aware true
dvc remote modify cloud-2 worktree true
git add .
git commit -m "initialized repo"

echo "Add data"
mkdir data
echo image1 > data/image1.png
echo image2 > data/image2.png
echo model > model.h5
dvc add data
dvc add model.h5
git add .
git commit -m "add data"

echo "Push data to default remote"
dvc push
git --no-pager diff
git commit -am "push data to default remote"

echo "Push data to other remote"
dvc push -r cloud-2
git --no-pager diff # Problem 1: overwrites all version_ids
git commit -am "push data to other remote"

echo "Push to different remotes per output"
echo "  remote: cloud-2" >> model.h5.dvc
dvc push
git --no-pager diff # Problem 2: pushes model.h5 to the default remote

echo "See model.h5 versions on remote 1"
aws s3api list-object-versions --bucket $BUCKET --prefix $REMOTE_1/model.h5
echo "See model.h5 versions on remote 2"
aws s3api list-object-versions --bucket $BUCKET --prefix $REMOTE_2/model.h5

Expected

1. DVC should keep track of the remote for each version_id, so that when pushing to a different remote, it appends a version_id for the other remote instead of overwriting it.
2. When using the remote-per-output remote: syntax in a .dvc file, DVC should push to that remote instead of the default.

[pmrowla]

Is the push/pull --remote use case actually something that we want to support? I can see the value in supporting multiple remotes per-repo via the explicit remote: per output syntax, and in this case we aren't actually supporting multiple remotes per output.

Supporting multiple remotes per-output leads to questionable behavior unless the user is always pushing everything to all of the remotes they intend to use, every time they make a modification to data in their DVC repo.

[pmrowla]

This is similar to the problem in #8298, where the existing behavior for push/pull --remote is actually pretty vague as to what the expected behavior (and intended use case) for using multiple remotes with DVC should actually be.

I think there's mainly 2 common use cases for having multiple remotes in DVC:

• Data segregation (i.e. certain data has to go in certain S3 buckets for privacy/security reasons)
• Read-only vs read/write access

---

In practice, the data segregation use case only works in DVC if the user is already using remote: in dvc.yaml/.dvc files to specify one remote per output, or if the user is manually pushing/pulling specific files and dirs each time with a specific --remote

dvc push --remote some-remote path/to/explicit/path

In this case, they are actually still only pushing each output to a single remote (per output), and should probably be using remote: in their dvc.yaml/.dvc file instead of using --remote in the CLI.

---

For the read-only vs read/write use case, it works in non-cloud versioning scenarios since you can do things like set up an S3 remote with write credentials, and then use the public https://... URL for that bucket as a default read-only remote.

For DVC's purposes, this is not possible to do in a cloud versioning scenario, unless your default read-only remote is still the s3://... bucket URL, but with actual S3 credentials that allow public reads. You cannot just point DVC to the https:// bucket URL to get a read-only remote in this scenario. What you would need to is to dvc push everything to your https:// remote, and get versioned URLs for that remote, which isn't possible (since you can't push to a public https: S3 URL)

[dberenbaum]

In practice, the data segregation use case only works in DVC if the user is already using remote: in dvc.yaml/.dvc files to specify one remote per output, or if the user is manually pushing/pulling specific files and dirs each time with a specific --remote

True, so for this use case, it may not be needed to append multiple version_id values for a single output. However, right now the remote: field isn't respected at all for cloud versioning AFAICT.

For DVC's purposes, this is not possible to do in a cloud versioning scenario, unless your default read-only remote is still the s3://... bucket URL, but with actual S3 credentials that allow public reads.

I think users may want private but still read-only cloud remotes (the bucket policy grants the user read permissions but not write permissions).

---

A 3rd use case is migrating remotes (let's say a company switches cloud providers). Without keeping track of the remote that generated each version_id, this seems like it will make it nearly impossible to keep using your existing Git repo since it will be full of version_id values that will throw errors when used on the new remote. At least DVC would know to ignore these if it were aware of the remote associated with them.

This info should probably be separate from the existing remote: field in .dvc/dvc.yaml (it belongs in dvc.lock and not dvc.yaml). We need to differentiate between configuration/declaration (defining which remote to use for push/pull operations) and state (where the output has been saved and what is its version ID there).

[dberenbaum]

Moving this to p1 since it will be a breaking change and keeps coming up

[pmrowla]

see also: #8653 (comment)

[dberenbaum]

Moving this to p0 since it blocks any public release of cloud versioning