import/get: improve message when we try to re-import an artifact

[shcheklein]

We need to check (and fail gracefully?) if someone tries to do import/get of an output from the imported stage.

Example:

mkdir test-import
cd test-import
git init
dvc init
dvc import https://github.com/iterative/example-get-started model.pkl
dvc remote add -d myremote /tmp/storage
git commit -a -m "add remote"
cd ..
mkdir test-recursive-import
git init
dvc init
dvc import ../test-import/ model.pkl

outputs:

Importing 'model.pkl (../test-import/)' -> 'model.pkl'
WARNING: Some of the cache files do not exist neither locally nor on remote. Missing cache files:
name: ../../../../private/var/folders/_1/dxrf7_f15sn4r01jvqvr5b6h0000gn/T/tmprcut42dgdvc-erepo/model.pkl, md5: 662eb7f64216d9c2c1088d0a5e2c6951
WARNING: Cache '662eb7f64216d9c2c1088d0a5e2c6951' not found. File 'model.pkl' won't be created.
ERROR: failed to import 'model.pkl' from '../test-import/'. - output 'model.pkl' does not exist

Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!

which is ugly and technically correct but can improved to warn that we are trying to re-import imported output which will always fail since we don't cache imports.

@shcheklein , do you have an example?

[shcheklein]

@MrOutis will do an example 👍 , should be quick and easy

[shcheklein]

mkdir test-import
cd test-import
git init
dvc init
dvc import https://github.com/iterative/example-get-started model.pkl
cd ..
mkdir test-recursive-import
dvc import ../test-import/ model.pkl
Importing 'model.pkl (../test-import/)' -> 'model.pkl'
ERROR: failed to import 'model.pkl' from '../test-import/'. - Failed to clone repo '../test-import/' to '/var/folders/_1/dxrf7_f15sn4r01jvqvr5b6h0000gn/T/tmpbrj429l0dvc-erepo': Cmd('git') failed due to: exit code(128)
  cmdline: git clone --no-single-branch -v ../test-import/ /var/folders/_1/dxrf7_f15sn4r01jvqvr5b6h0000gn/T/tmpbrj429l0dvc-erepo
  stderr: 'fatal: repository '../test-import/' does not exist
'

looks like a separate bug to me. Import does not work with relative path?

Looks like it's a git specific behavior:

git clone --no-single-branch -v ../test-import/ /var/folders/_1/dxrf7_f15sn4r01jvqvr5b6h0000gn/T/tmp5qo0ukuzdvc-erepo

does not work

[shcheklein]

mkdir test-import
cd test-import
git init
dvc init
dvc import https://github.com/iterative/example-get-started model.pkl
dvc remote add -d myremote /tmp/storage
git commit -a -m "add remote"
cd ..
mkdir test-recursive-import
git init
dvc init
dvc import ../test-import/ model.pkl

outputs:

Importing 'model.pkl (../test-import/)' -> 'model.pkl'
WARNING: Some of the cache files do not exist neither locally nor on remote. Missing cache files:
name: ../../../../private/var/folders/_1/dxrf7_f15sn4r01jvqvr5b6h0000gn/T/tmprcut42dgdvc-erepo/model.pkl, md5: 662eb7f64216d9c2c1088d0a5e2c6951
WARNING: Cache '662eb7f64216d9c2c1088d0a5e2c6951' not found. File 'model.pkl' won't be created.
ERROR: failed to import 'model.pkl' from '../test-import/'. - output 'model.pkl' does not exist

Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!

which is ugly and technically correct but can improved to warn that we are trying to re-import imported output which will always fail since we don't cache imports.

[kiranvinodcgs]

I would like to work on this issue

[kurianbenoy]

Can't reproduce this issue @shcheklein

[efiop]

@shcheklein We could totally import recursive outs as well, it just requires some refactoring. But let's forbid it for now, until we get a request for that.

@kiranvinodcgs Sure! So to clarify, to fix this you need to put a check here https://github.com/iterative/dvc/blob/0.66.3/dvc/dependency/repo.py#L75 , that would look something like:

if out.stage.is_repo_import:
    raise DvcException("Recursive imports are currently unsupported.")

[efiop]

@kurianbenoy Indeed, the script provided by @shcheklein doesn't work. Here is a modified script:

#!/bin/bash
set -e
set -x

rm -rf test-import
mkdir test-import
pushd test-import
git init
dvc init
dvc import https://github.com/iterative/example-get-started model.pkl
dvc remote add -d myremote /tmp/storage
git add .
git commit -a -m "add remote"
popd

rm -rf test-recursive-import
mkdir test-recursive-import
cd test-recursive-import
git init
dvc init
dvc import ../test-import/ model.pkl

[shcheklein]

@kurianbenoy @efiop sorry, guys! 🙏 will be more careful next time!