filesystem_spec
filesystem_spec copied to clipboard
fsspec
A specification that python filesystems should adhere to.
First of all, thanks for your great work. I have been uning FSSpec for a while and it is a great piece of software. I had a question regarding the...
When opening a zstd compressed reference file using: fs = fsspec.filesystem("reference", fo='s3://esip-qhub-public/ecmwf/ERA5_2020_2022_multivar.json.zst', ref_storage_args= {"compression": "zstd"}, remote_protocol='s3', remote_options={'anon':True}) I get the error: Traceback (most recent call last): File "/home/peter/Documents/intake_test/test_local.py", line 3,...
When using a mapper on a local filesystem, it's possible to create nested subdirectory structures using `FSMap` as in: ```python import fsspec import tempfile with tempfile.TemporaryDirectory() as d: m =...
**In all glob implementations, "x/\**/y" should also match "x/y" as a default behavior** (see [glob doc](https://docs.python.org/3/library/glob.html) for python or [glob.js](https://github.com/isaacs/node-glob#readme)). **However, it is not the case in fsspec** ("/**/" is...
Config file based options for passing specific arguments to specific backends don't work for non-string parameters. For example, from the wording of #453 if I want to pass a custom...
```python from fsspec.implementations.local import LocalFileSystem fs = LocalFileSystem() fs.ls("C:/") ``` This lists the content of os.getcwd() instead of C:/. The same for similar commands like ``` fs.ls("file://C:/") ``` pretty sure...
Ref: https://github.com/fsspec/filesystem_spec/issues/748 which I found while also wondering about `AttributeError: 'SFTPFile' object has no attribute 'blocksize'` error ;-) I guess there is no "sparse" cache due to not implemented "range"...
Caching S3 data with e.g., `simplecache` works fine. However, S3FS also supports [versioned buckets](https://s3fs.readthedocs.io/en/latest/#bucket-version-awareness). When a `version_id` is provided to `open`, using a cache leads to incorrect behavior, as the...
I was recently reminded that `file:` URIs can specify a hostname. If a `file:` URI is passed with a hostname, it is currently dropped on the floor silently. Should something...
