For a large dataset `dvc add` creates symlinks to wrong files in the cache

[maxme1]

Bug Report

I have a large dataset (~60GB) of mostly large files. Each file has additional json-files associated with it.
After I use dvc add FOLDER all files are replaced with symlinks, and some of them are referencing wrong files in the cache (but from the same dataset). Moreover most of the links reference the same file, so the links are not just shuffled.

I tried to reproduce this issue with json-files only, but didn't succeed.
However the issue persists if I include the large files, even though it's somewhat random: each time (running add) the links point to different files.

Information about my setup

Output of dvc version:

$ dvc version

DVC version: 1.2.2
Python version: 3.6.8
Platform: Linux-4.15.0-107-generic-x86_64-with-debian-stretch-sid
Binary: False
Package: pip
Supported remotes: http, https, s3, ssh
Filesystem type: ('nfs4', EDITED_OUT_THE_PATH)

Additional Information:

The cache is stored outside of the repository on an NFS4-mounted disk.

[efiop]

Hi @maxme1 ! That sounds terrible! 😱

Moreover most of the links reference the same file, so the links are not just shuffled.

Is there anything specific about that file? Is it an empty one?

Also, have you tried copy or hardlink? Are those affected?

[maxme1]

Hi @efiop !

Is there anything specific about that file? Is it an empty one?

No, just some random file from all the json-files.
I forgot to mention that some json-files have identical content (basically they contain the shape of an associated numpy array).
None of the files are empty. Most of them have the same length (in bytes).

Also, have you tried copy or hardlink? Are those affected?

No, but I'll give it a try.

[efiop]

I forgot to mention that some json-files have identical content (basically they contain the shape of an associated numpy array).

@maxme1 You mean that they have identical content originally, right? Before you've dvc-added them? If so, all of those would be linked to the same cache file, because cache is content-based.

[maxme1]

You mean that they have identical content originally, right?

Not quite. Here is a sample without the large files (only json).
For example often almost all spacing.json files are getting mapped to a single shape.json file, even though their contents are always different.

[efiop]

@maxme1 Interesting. Ok, let's see if this happened during collection or during linking. In the *.dvc file that was created by dvc add, there is an md5 that looks something like 12345.dir. Using it, you could find a .dvc/cache/12/345.dir json file, that contains list of files that dvc collected and their respective hashes. Can you check whether those 2 files that shouldn't have the same content have the same hash in that *.dir file? If they do - something was wrong in the original dataset or something went wrong during collection. If they don't - then something wen't wrong during linking.

[maxme1]

I remounted the drive on which the cache is located, and rebooted the machine. Now everything seems to work fine. I suppose the problem was with NFS. Hope it will be useful if someone encounters the same problems.

I guess the issue can be closed now. Thanks for your time, @efiop !

[efiop]

@maxme1 Whoa, that's a terrible bug to have 😱 Btw, could you share more details about your setup? Like what nfs version is used? Is that for NAS or something else?

[maxme1]

Yup, I'm still not sure what was the real cause of the problem.
My setup: several machines with NFS4-mounted HDDs, the cache is located on one of the shared drives.
The machines share a local network, no NAS is used.

[efiop]

@maxme1 If you run into this issue again, please let us know. Closing for now.