filesystem_spec
filesystem_spec copied to clipboard
Transaction doesn't work well with compression
Here is the code to reproduce (fsspec version 2024.3.1):
import fsspec
fs = fsspec.filesystem('local')
with fs.transaction:
with fs.open('/tmp/a.txt.gz', 'wt', compression='infer') as f:
f.write('Hello, world!\n')
Here is the error:
Traceback (most recent call last):
File "/tmp/a.py", line 5, in <module>
with fs.transaction:
File "/home/user/anaconda3/envs/lib/python3.11/site-packages/fsspec/transaction.py", line 28, in __exit__
self.complete(commit=exc_type is None)
File "/home/user/anaconda3/envs/lib/python3.11/site-packages/fsspec/transaction.py", line 44, in complete
f.commit()
^^^^^^^^
AttributeError: 'GzipFile' object has no attribute 'commit'
Changing the file name from /tmp/a.txt.gz
to /tmp/a.txt
works.
Transactions don't work with any file-like objects which we don't define ourselves in this repo. We could in principle wrap all the ones like GzipFile as we do with fsspec.implementations.local.LocalFileOpener for local files; indeed, this could maybe be phrased as a class decorator.