filesystem_spec icon indicating copy to clipboard operation
filesystem_spec copied to clipboard

copy directory contains symbolic to subdir to directory twice bigger

Open pyfreyr opened this issue 7 months ago • 11 comments

src dir:

data/2025-05-16
├── sparse_embeddings -> model/sparse_embeddings
└── model/sparse_embeddings/

when use local fs to copy, the sparse_embeddings symbolic link will copy model/sparse_embeddings data duplicated. As a result, the data/2025-05-16 directory will be twice bigger than origin.

from fsspec.implementations.local import LocalFileSystem

fs = LocalFileSystem()
fs.copy("data/2025-05-16/", "models/2025-05-16/", recursive=True)

pyfreyr avatar May 19 '25 05:05 pyfreyr