filesystem_spec icon indicating copy to clipboard operation
filesystem_spec copied to clipboard

rsync: Modeled after fsspec.generic.rsync this uses two fs explicitly

Open f4hy opened this issue 1 year ago • 1 comments

Addressing #1398

Pretty much identical to the generic one but doesn't create the directories async. The generic one defines a make_many_dirs which tries to make them async if it can and falls back to serial, this just uses the simpler version of non async. Can do the same helper function though.

This is needed for cases of syncing two different storage systems that might have the same protocol. An example would be two different s3 buckets that use different credentials. This allows you to make two different s3fs instances each passing different parameters and still sync them.

The generic interface is great for giving a simple implicit way to sync things, but often we need fine grained control of what our filesystem parameters are so having an explicit method as well is useful. (Zen of Python line 2)

f4hy avatar Oct 24 '23 20:10 f4hy

NM this is not ready yet. The copy logic needs to be ported over as well. will finish this tomorrow sorry.

f4hy avatar Oct 24 '23 21:10 f4hy