fsspec
Full support for decompressive transcoding (e.g. Content-Encoding: gzip ) during reading
This fixes #461 and #233 without needing users to change their existing code.
The bugs were caused by assumptions in fsspec about the veracity and non-ambiguity of 'size' information returned by AbstractBufferedFile subclasses like GCSFile and AbstractFileSystem subclasses like GCSFileSystem (e.g. self.size = self.details["size"] in AbstractBufferedFile, which is used by all base caches to truncate requests and responses). Since in GCS if compression-at-rest/compression transcoding is used there's no way to retrieve the real size of the object's content without decompressing the whole thing either server or client side, fixing these issues required overriding some behaviors in the underlying base classes. Care was taken to preserve behavior for storage objects not using compression at rest, however.
The fix keeps the data handling for GCS files which do not use compression at rest/compressive transcoding mostly identical by adding new control flow to detect when transcoding is done and adding some branch logic for handling those edge-cases. This did unfortunately mean implementing implementing variant methods to base classes with only minor changes to how they perform underlying operations (e.g. read() in GCSFile) which were previously just inherited from AbstractBufferedFile.
It does introduce one new semantic to GCSFs. In line with fsspec's ArchiveFileSystem semantics, GCSFile will return size = None when the file can not be determined fully in advance. This allows us to distinguish known zero size and unknown size, but on the plus side this also means end-users have greater guarantees that sizes they get back are meaningful.
The only new performance overhead seen by non-users of compressive decoding is a single info() call resulting in a HEAD request done before the point where we create the GCSFile object in GCSFilesystem on file open()s, because we need to swap out the cache to one compatible with the lack of concrete file size but do not yet have the information to make that control flow decision. We can probably switch to the new transcoding cache wholesale in the future when we have faith it holds up to eliminate this call though, but I made it work this way to keep the data and control flow the same for the base case where users are not using compressive transcoding.
Since the compression-at-rest case was completely broken, doing it this way means that even if these changes end up totally disastrous (they shouldn't though) they should only break things which were already broken and therefore this pull request should be very low risk.
As an aside, this is my first pull request on GitHub to a broader community project, so let me know if there's anything I can do to polish this up.
Before we merge I think we should add some unit tests for the transcoding case, the new cache and maybe beef up some test cases in the non-transcoding case (e.g. normally uploaded gzip files) to ensure we don't introduce some new issue or cause regressions in the future, but I could use some help in understanding how to develop for the test suite here, test them locally, test them against whatever dev buckets there are, etc. I'm familiar with pytest though.
There is quite a lot of code here, and I haven't yet had a chance to look through it, sorry.
As an aside, this is my first pull request on GitHub to a broader community project, so let me know if there's anything I can do to polish this up.
Welcome, and thanks for getting involved.
One comment on caching: only readahead and none caching make sense, since we can only really stream from the start when using gzip.
Question:
- when you issue bytes= range requests for compression encoded files, is this evaluated against the on-disk file? I assume, then, that anything not starting at zero would fail to decompress within aiohttp.
- is it possible to turn decompression off in the client, so that fsspec can do it (e.g.,
fsspec.open("gcs://...", compression="gzip")works fine for files that happen to be compressed, but without explicit transcoding set)?
I gotta look back into this a bit more extensively. I've noticed many subtle and not so subtle issues with my attempt at solving this and have to go back to the drawing board.
Thanks for your patience