filesystem_spec icon indicating copy to clipboard operation
filesystem_spec copied to clipboard

ArrowFSWrapper should not use "/" root_marker for all filesystems

Open b-phi opened this issue 1 year ago • 1 comments

For the arrow S3FileSystem for example, using root_market = "/" throws an error when trying to upload a local file. Love the arrow integration, please let me know if I'm misusing something here.

from pyarrow.fs import S3FileSystem
from fsspec.implementations.arrow import ArrowFSWrapper

fs = ArrowFSWrapper(S3FileSystem())
fs.upload('foo.txt', 'bucket/folder/foo.txt')

Traceback (most recent call last): File "", line 1, in File ".../fsspec/spec.py", line 1530, in upload return self.put(lpath, rpath, recursive=recursive, **kwargs) File ".../fsspec/spec.py", line 1057, in put self.put_file(lpath, rpath, **kwargs) File ".../fsspec/spec.py", line 984, in put_file self.mkdirs(self._parent(os.fspath(rpath)), exist_ok=True) File ".../fsspec/spec.py", line 1498, in mkdirs return self.makedirs(path, exist_ok=exist_ok) File ".../fsspec/implementations/arrow.py", line 22, in wrapper return func(*args, **kwargs) File ".../fsspec/implementations/arrow.py", line 193, in makedirs self.fs.create_dir(path, recursive=True) File "pyarrow/_fs.pyx", line 603, in pyarrow._fs.FileSystem.create_dir File "pyarrow/error.pxi", line 91, in pyarrow.lib.check_status pyarrow.lib.ArrowInvalid: Path cannot start with a separator ('/bucket/folder')

If I set root_market to '' manually, the upload succeeds

fs.__class__.root_marker = ''
fs.upload('foo.txt', 'bucket/folder/foo.txt')

b-phi avatar Dec 12 '23 21:12 b-phi

root_marker is a class attribute and (I think) only used in class methods, so I'm not sure how we can make it dependent on the instance being wrapped.

I don't suppose the following would have solved your problem

fs.root_marker = ""

I am open to suggestions!

martindurant avatar Dec 13 '23 15:12 martindurant