filesystem_spec icon indicating copy to clipboard operation
filesystem_spec copied to clipboard

Transaction doesn't work well with compression

Open roosephu opened this issue 10 months ago • 1 comments

Here is the code to reproduce (fsspec version 2024.3.1):

import fsspec

fs = fsspec.filesystem('local')

with fs.transaction:
    with fs.open('/tmp/a.txt.gz', 'wt', compression='infer') as f:
        f.write('Hello, world!\n')

Here is the error:

Traceback (most recent call last):
  File "/tmp/a.py", line 5, in <module>
    with fs.transaction:
  File "/home/user/anaconda3/envs/lib/python3.11/site-packages/fsspec/transaction.py", line 28, in __exit__
    self.complete(commit=exc_type is None)
  File "/home/user/anaconda3/envs/lib/python3.11/site-packages/fsspec/transaction.py", line 44, in complete
    f.commit()
    ^^^^^^^^
AttributeError: 'GzipFile' object has no attribute 'commit'

Changing the file name from /tmp/a.txt.gz to /tmp/a.txt works.

roosephu avatar Apr 19 '24 10:04 roosephu

Transactions don't work with any file-like objects which we don't define ourselves in this repo. We could in principle wrap all the ones like GzipFile as we do with fsspec.implementations.local.LocalFileOpener for local files; indeed, this could maybe be phrased as a class decorator.

martindurant avatar Apr 25 '24 15:04 martindurant