filesystem_spec
filesystem_spec copied to clipboard
fsspec
A specification that python filesystems should adhere to.
Setting protocol specific options has been a convenient method for overriding the default options for each protocol. E.g., the Azure blob storage implementation behaves peculiarly and requires setting `anon=False` to...
While new `fsspec`-compatible filesystem can easily be registered via `fsspec.specs` entry points, there is no counterpart for compression types. As we already have the `fsspec.compression.register_compression` public API for custom compression...
If the filesystem provides files with same name at different directory then it breaks for example with HTTPFilesystem if the response is as follows: ``` "/folder1/file" "/folder2/file" ``` Then it...
This might be a naive question but I have spent a bit of time trying to figure it out and haven't made much progress. I'm trying to do this workflow...
Currently the listing APIs only return the listings when everything is read (e.g `ls()` / `find()` / `glob()`) though this is blocking applications which normally would be able to process...
## Background `HTTPFileSystem._open` chooses between `HTTPFile` or `HTTPStreamFile`, and only the former allows random access, by using the `"Range"` header in subsequent HTTP GET requests. Servers do not always respond...
A lot of backends can throw exceptions that are swallowed by fsspec, we should at least log these errors in debug or info level using the builtin loggers with exc_info=True...
This is a similar issue to #924. **Code** ```python fs = DirFileSystem(f'/{bucket_name}', S3FileSystem()) files = fs.ls('/') ``` **Error** ```text Traceback (most recent call last): File "/Users/ppatterson/src/b2_zip_files/app.py", line 40, in files...
Hi Fsspec experts, I'm presently using Fsspec with one of the built in caching file systems, `WholeFileCacheFileSystem`, and enjoy using the callback feature that is present in functions like `.get()`...
``` from fsspec.core import url_to_fs ufs, url = url_to_fs("https://example.org/") f = ufs.open("https://example.org/no-range-requests/data.txt") f.seek(8192) f.read(1) ``` Will raise `# ValueError: The HTTP server doesn't appear to support range requests.` I was...
