filesystem_spec
filesystem_spec copied to clipboard
rsync: Modeled after fsspec.generic.rsync this uses two fs explicitly
Addressing #1398
Pretty much identical to the generic one but doesn't create the directories async. The generic one defines a make_many_dirs which tries to make them async if it can and falls back to serial, this just uses the simpler version of non async. Can do the same helper function though.
This is needed for cases of syncing two different storage systems that might have the same protocol. An example would be two different s3 buckets that use different credentials. This allows you to make two different s3fs instances each passing different parameters and still sync them.
The generic interface is great for giving a simple implicit way to sync things, but often we need fine grained control of what our filesystem parameters are so having an explicit method as well is useful. (Zen of Python line 2)
NM this is not ready yet. The copy logic needs to be ported over as well. will finish this tomorrow sorry.