fsspec backend

[majidaldo]

I really think dvc should have this backend. There is much history to this filesystem abstraction so it could leverage the backends for fsspec.

[pmrowla]

fsspec looks like an interesting project, thanks for the suggestion!

We already use a similar abstraction internally in DVC, and I'm not sure how much we would actually gain by switching to an alternative implementation. Is there a particular remote FS type that is supported in fsspec that you are looking for in DVC?

[efiop]

We have considered it previously while working on pyarrow (it uses fsspec too), but for now decided that it is not really worth refactoring so much stuff just for the sake of it. It will probably happen in the future, once we are ready for it and have time for it.

[efiop]

For the record, we are indeed going in fsspec direction in our internals. Current roadmap is

• make sure no Tree uses self.path_info(self._path in case of gdrive) outside of __init__. We've done some work there already, with only gdrive being the last blocker right now. Tree.path_info should really move into something like cache_dir in a corresponding CloudCache(they currently use self.tree.path_info).
• make our Trees __init__ only accept their configuration options, don't pass repo or path_info(if path_info has user/port/etc, these should be extracted and passed directly to the tree's __init__)
• make our tree methods only accept strings as paths, not path_infos (or maybe make it handle both, but strings/bytes are better when dealing with many-many objects, which is currently a problem for us

  dvc/dvc/cache/base.py

  Line 103 in 47bdf52

  # Prefer string path over PathInfo when possible due to performance

also somewhere after the first step we can rename our Tree/tree to FileSystem/FS/fs/etc so it is less confusing.

[Nantero1]

Hey, do you have any idea which fsspec methods (mkdir, ls, info, rmdir,...) you require and which are a nice-to-have? I wrote a Nexus3 integration for DVC and thinking about migrating it to fsspec so it can be one day used with DVC by the community.

[efiop]

@BenLinnik There is no formal list yet, but you can see all required methods in https://github.com/iterative/dvc/blob/master/dvc/fs/azure.py , for example. (e.g. see all self.fs.* calls).

[Nantero1]

@BenLinnik There is no formal list yet, but you can see all required methods in https://github.com/iterative/dvc/blob/master/dvc/fs/azure.py , for example. (e.g. see all self.fs.* calls).

Seems straightforward. Nothing too fancy. I could do that. When looking at the azure code, I wonder. Why don't you use a fs method in the self._upload_fobj method?

[efiop]

@BenLinnik We do, through fs.open. But if there is a more suitable method - we can use it too.

[Nantero1]

@BenLinnik We do, through fs.open. But if there is a more suitable method - we can use it too.

Ah sorry, you are absolutely right. The fs.open is in self.open.

A great idea to move to fsspec. This will make it possible to integrate dvc with any fsspec supported framework. That's a clever architectural choice, despite it makes my PR obsolete at the moment 👍

[martindurant]

(fsspec admin here)

I just noticed this issue, and am glad to see the energy, and the appearance of an fsspec backend in your API, alongside your own implementations. Please don't hesitate to help us improve every data user's experience!

Did you know that fsspec is interested in exposing DVC as a filesystem backend? In https://github.com/intake/filesystem_spec/blob/master/fsspec/implementations/dvc.py is some nascent code which is not references in the official docs or tested. It has been a while since I looked at that code, so it may not do anything useful by now. However, I do think it's generally useful to have a simple abstraction over-layer to DVC in fsspec, even if that ends up calling other fsspec backends internally when fetching files.
Indeed, in some ways, DVC could act like a mix of a cache layer and git.

[efiop]

Hi @martindurant !

Thanks a lot for creating and maintaining fsspec! 🙏 Was meaning to send a similar introductory message to you today 😉

We were inventing something similar ourselves with our legacy Tree classes, but then stumbled upon fsspec and were impressed by how thought through the spec is. Really wish fsspec existed in the early days of DVC 😃

Thanks to great work by @isidentical we were able to temporarily wrap and use gcsfs and adlfs in dvc and we are looking forward to migrating to s3fs and others. We are heavy users of these filesystems, so we've been catching some bugs and contributing some fixes to the upstream already (or at least reporting them in some cases). Even more to come 🙂

Did you know that fsspec is interested in exposing DVC as a filesystem backend?

We've had DvcTree and RepoTree(aka Dvc/RepoFileSystem in dvc/fs/) for a long time now and it was a surprise to find existing fsspec implementation for it! It might make sense to just register our DvcFileSystem implementation instead of the existing experimental one, so we take the burden of maintaining it from you and also provide the most feature-complete and supported implementation. Let us know what you think about that. Of course, our DvcFileSystem is not ready right now, but we are working on making it comply with fsspec.