dvc icon indicating copy to clipboard operation
dvc copied to clipboard

localfs: make it fsspec-compatible

Open skshetry opened this issue 3 years ago • 5 comments
trafficstars

The LocalFileSystem is the only filesystem that is not fsspec-compatible. There's also DvcFileSystem and RepoFileSystem but they are virtual filesystem and needs LocalFileSystem to be first compatible.

https://github.com/iterative/dvc/blob/9b5ccb75549ab950ece23b7938ea0eea637c440d/dvc/fs/local.py#L15

This may help us use localfs in more locations instead of os.path/open functions. And also may get rid of the ugly inheritance chain with FileSystem/FSSpecWrapper.

skshetry avatar Feb 07 '22 07:02 skshetry

This may help us use localfs in more locations instead of os.path/open functions.

Is there any idea of what this will mean for performance?

dberenbaum avatar Feb 07 '22 20:02 dberenbaum

@skshetry Is this something you plan to take or research? Should it be on the project board?

dberenbaum avatar Feb 14 '22 21:02 dberenbaum

Is there any idea of what this will mean for performance?

We already use localfs on data-management systems or wherever we are filesystem agnostic.

Is this something you plan to take or research? Should it be on the project board?

I noticed this while working on #7331, but it's not required right now.

skshetry avatar Feb 15 '22 06:02 skshetry

Can we lower the priority?

dberenbaum avatar Feb 16 '22 00:02 dberenbaum

@skshetry What's the status of this issue?

dberenbaum avatar Sep 19 '22 17:09 dberenbaum