Recursive directory copy between different filesystem

[hadim]

Is there is a way to recursively copy a directory between one file system and another? It seems like put is only restricted to local -> fs but not fs -> fs.

If it's not possible, I was thinking of doing a recursive ls followed by a simple loop on all the files and folders with mkdir and a custom copy_files function.

When doing the recursive ls, the path name does not contain the prefix of the fs. I am trying the following function to prefix a given path with its protocol, but I feel like it's going to be error-prone:

def prefix_with_protocol(path: str, fs: fsspec.AbstractFileSystem):
    """Given a filesystem, prefix a path with the protocol.
    Args:
        path: The path to prefix.
        fs: The filesystem to prefix with.
    """
    protocol = get_protocol(path, fs=fs)
    if protocol != "file":
        return f"{protocol}://{path}"
    return path

Is there is a utility function in fsspec to achieve something similar?

[martindurant]

The PR #828 is working on this kind of functionality, but it might take some time to arrive.

is there is a utility function in fsspec to achieve something similar?

You want fsspec.utils._ustrip_protocol, which will become a file system method in #828 .

[rom1504]

ah I see I just created a duplicate at https://github.com/fsspec/filesystem_spec/issues/909 and maybe you'll be interested by my solution to this problem @hadim (see in issue)