Bug: fsspec.put(..., recrusive=True) 2021.11.0

[peterdudfield]

Bug

I think I found a bug with the '.put' when copying from local to local. This is when using the ''recrusive" flag My aim is to copy the content of one folder to another

setup

This behaviour did not happen in 2021.07.0, but it did for 2021.11.0 This was on a mac

minimum example

I written a small example below, that can easily be made into a test.

import tempfile
import fsspec
import os
def test_put_files_recursive():
    """Check 'put' works"""

    print(fsspec.__version__)

    file1 = "test_file1.txt"
    file2 = "test_dir/test_file2.txt"

    with tempfile.TemporaryDirectory() as local_path, tempfile.TemporaryDirectory() as dst_path:

        filesystem = fsspec.open(local_path).fs

        # add fake file to dir
        path_and_filename_1 = os.path.join(local_path, file1)
        with open(path_and_filename_1, "w"):
            pass

        # add fake file to dir
        os.mkdir(f"{local_path}/test_dir")
        _ = os.path.join(local_path, file2)
        with open(os.path.join(local_path, file2), "w"):
            pass

        # no files in folder before folder
        assert len(filesystem.ls(dst_path)) == 0

        filesystem.put(local_path + "/", dst_path + "/", recursive=True)

        new_files = filesystem.ls(dst_path)
        assert len(new_files) == 2

test_put_files_recursive()

Can any help explain this, or if Im using the it incorrectly. .

[martindurant]

As far as I can tell, the right thing happened: when the destination directory already exists, you are indicating that you want to copy files and folders within the destination. The same would happen on the command line. If you do

filesystem.find(dst_path)

you will see that the files are nested within a folder whose name is the same as the basename of local_path.

[peterdudfield]

and perhaps 'copy' would do what I want it do, i.e copy the files within a folder, to another folder

[peterdudfield]

If so, I'm happy for the issue to be closed

[ianthomas23]

As @martindurant said, this behaviour is intended. We have checked cp, get and put and fixed where necessary to make them consistent with command-line cp. The issue with the comprehensive test script is #1062 and the PR that fixes it and demonstrates that the behaviour is correct is #1148.