filesystem_spec
filesystem_spec copied to clipboard
copy directory contains symbolic to subdir to directory twice bigger
src dir:
data/2025-05-16
├── sparse_embeddings -> model/sparse_embeddings
└── model/sparse_embeddings/
when use local fs to copy, the sparse_embeddings symbolic link will copy model/sparse_embeddings data duplicated. As a result, the data/2025-05-16 directory will be twice bigger than origin.
from fsspec.implementations.local import LocalFileSystem
fs = LocalFileSystem()
fs.copy("data/2025-05-16/", "models/2025-05-16/", recursive=True)