Skip to content

Unable to outpus directly to google cloud storage like dvc v1.11.6 #5854

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
dpcabo opened this issue Apr 20, 2021 · 1 comment
Closed

Unable to outpus directly to google cloud storage like dvc v1.11.6 #5854

dpcabo opened this issue Apr 20, 2021 · 1 comment

Comments

@dpcabo
Copy link

dpcabo commented Apr 20, 2021

Bug Report

Description

Whe I try to run this stage y got this error but the file exists.
This error happens only with dvc 2.X, if I downgrade dvc to the version 1.11.6 this stage runs properly.

Reproduce

  1. Create a remote bucket in google cloud "mybucket"
  2. Create a local file "touch delete.me"
  3. Run dvc stage: dvc run -n mystage -d delete.me --external -o gs://mybucket/delete.me gsutil cp delete.me gs://mybucket/

Expected

The same behaviour than the goten with dvc 1.11.6

Environment information

Output of dvc doctor:

$ dvc doctor

Platform: Python 3.9.1 on Linux-5.4.0-1037-gcp-x86_64-with-glibc2.31
Supports: All remotes
Cache types: <https://error.dvc.org/no-dvc-cache>
Caches: local, gs
Remotes: gs, gs, gs
Workspace directory: ext4 on /dev/root
Repo: dvc, git
@efiop
Copy link
Contributor

efiop commented Apr 20, 2021

Hi @dpcabo

We've removed support for gs because it was fundamentally broken for multipart objects. External output scenario is highly experimental in general and we don't recommend using it unless you really need to. But we are aware that there is a demand for that, so it is on our todo list to research it deeper and maybe propose a better general solution (related #3920).

Just to be clear, we are not going to support gs external outputs the same way anymore, so unfortunately you might be stuck with older dvc for now, if you have to use that feature.

Btw, could you describe your scenario, so we could understand better your particular case and maybe suggest an alternative?

@efiop efiop closed this as completed Apr 20, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants