filesystem_spec
filesystem_spec copied to clipboard
ArrowFSWrapper should not use "/" root_marker for all filesystems
For the arrow S3FileSystem for example, using root_market = "/"
throws an error when trying to upload a local file. Love the arrow integration, please let me know if I'm misusing something here.
from pyarrow.fs import S3FileSystem
from fsspec.implementations.arrow import ArrowFSWrapper
fs = ArrowFSWrapper(S3FileSystem())
fs.upload('foo.txt', 'bucket/folder/foo.txt')
Traceback (most recent call last): File "
", line 1, in File ".../fsspec/spec.py", line 1530, in upload return self.put(lpath, rpath, recursive=recursive, **kwargs) File ".../fsspec/spec.py", line 1057, in put self.put_file(lpath, rpath, **kwargs) File ".../fsspec/spec.py", line 984, in put_file self.mkdirs(self._parent(os.fspath(rpath)), exist_ok=True) File ".../fsspec/spec.py", line 1498, in mkdirs return self.makedirs(path, exist_ok=exist_ok) File ".../fsspec/implementations/arrow.py", line 22, in wrapper return func(*args, **kwargs) File ".../fsspec/implementations/arrow.py", line 193, in makedirs self.fs.create_dir(path, recursive=True) File "pyarrow/_fs.pyx", line 603, in pyarrow._fs.FileSystem.create_dir File "pyarrow/error.pxi", line 91, in pyarrow.lib.check_status pyarrow.lib.ArrowInvalid: Path cannot start with a separator ('/bucket/folder')
If I set root_market
to ''
manually, the upload succeeds
fs.__class__.root_marker = ''
fs.upload('foo.txt', 'bucket/folder/foo.txt')
root_marker is a class attribute and (I think) only used in class methods, so I'm not sure how we can make it dependent on the instance being wrapped.
I don't suppose the following would have solved your problem
fs.root_marker = ""
I am open to suggestions!