filesystem_spec icon indicating copy to clipboard operation
filesystem_spec copied to clipboard

UnsupportedOperation error with ZipFileSystem using append mode

Open forman opened this issue 1 year ago • 4 comments

fsspec.implementations.zip.ZipFileSystem seems to have an issue with append mode 'a'. I assume, opening in mode 'a' allows for adding new Zip-entries to an existing Zip archive. At least this is what I need. While opening in append mode first seems to work - I can write to the archive - it will fail once the filesystem is closed.

While using mode 'w' works and creates an empty archive like so:

fs = fsspec.filesystem("zip", fo="test-w.zip", mode="w")
# write stuff here or do nothing...
fs = None  # <-- force garbage collection of fs object

An error is raised when using mode 'a' once the filesystem is auto-closed by setting the reference to None or by calling fs.close() directly:

fs = fsspec.filesystem("zip", fo="test-a.zip", mode="a")
# write stuff here or do nothing...
fs = None   # <-- error, also for fs.close()

Traceback:

Exception ignored in: <function ZipFileSystem.__del__ at 0x7fa07d5e4fe0>
Traceback (most recent call last):
  File ".../lib/python3.11/site-packages/fsspec/implementations/zip.py", line 73, in __del__
    self.close()
  File ".../lib/python3.11/site-packages/fsspec/implementations/zip.py", line 78, in close
    self.zip.close()
  File ".../lib/python3.11/zipfile.py", line 1892, in close
    self._write_end_record()
  File ".../lib/python3.11/zipfile.py", line 1989, in _write_end_record
    self.fp.truncate()
io.UnsupportedOperation: truncate

Reproducible on Linux and Windows.

Note, Python docs on zipfile.ZipFile state: If mode is 'a' and the file does not exist at all, it is created. If mode is 'r' or 'a', the file should be seekable.

forman avatar Dec 06 '23 11:12 forman

I believe fsspec.implementations.local.LocalFileOpener may need to explicitly implement truncate().

martindurant avatar Dec 06 '23 14:12 martindurant

I believe it should also be seekable if zipfile.ZipFile requires random-access (maybe in append mode only).

forman avatar Dec 06 '23 18:12 forman

.seekable() is already explicitly implemented.

martindurant avatar Dec 06 '23 18:12 martindurant

@martindurant , thanks! :)

forman avatar Dec 09 '23 13:12 forman