filesystem_spec
filesystem_spec copied to clipboard
fsspec
A specification that python filesystems should adhere to.
Currently, it's hard in fsspec to deregister/close callbacks, especially branched/nested ones after one `put_file`/`get_file` operation is done. In dvc, we are trying to do batched uploads for better performance, which...
This warning appears in tests: ``` fsspec/implementations/tests/test_hdfs.py::test_put_file_get_file /home/runner/work/filesystem_spec/filesystem_spec/fsspec/implementations/tests/test_hdfs.py:16: FutureWarning: pyarrow.hdfs.HadoopFileSystem is deprecated as of 2.0.0, please use pyarrow.fs.HadoopFileSystem instead. hdfs = pyarrow.hdfs.HadoopFileSystem() ``` According to the pyarrow [release history](https://pypi.org/project/pyarrow/#history), current...
One of the things that have come up when trying to integrate fsspec into tensorboard (https://github.com/tensorflow/tensorboard/pull/5248) is that there aren't any standard path operations as part of fsspec as far...
Idea: to make a method which works something like a low-featured version of rsync, to make local and remote sets of files equivalent (normal filesystem) or between filesystems (generic filesystem)....
Since I introduced `read_text`/`write_text` with `newline=` parameters, I was trying to implement the async versions of it. Since asyncfs don't have an async version of `open`, for ease, I was...
Using a ssh filesystem and put_files gives an error ```python import fsspec import sshfs fs = fsspec.filesystem("ssh", host='xx.xx.xx.xx', username='eouser') fs.put_file(lpath="t", rpath="/home/me/t") ``` I receive an exception: ``` File "/home/me/miniconda3/envs/olci/lib/python3.9/site-packages/fsspec/callbacks.py", line...
We often have this pattern: ``` def get_filesystem(path: str, **kwargs: Any) -> fsspec.AbstractFileSystem: """Returns the appropriate filesystem to use when handling the given path.""" fs, _ = url_to_fs(path, **kwargs) return...
Hello, I'm using fsspec with STAC metadata as a way to download assets in STAC Items in a standard way. These assets are most commonly http or s3 but could...
https://github.com/koenvo/pyodide-http provides patches to requests and urllib, to enable python stack HTTP calls in the browser. Since several fsspec backends (http, s3, gcs. azure) depend on HTTP, this raises the...
`cifar-10-python.tar`: ```python tfs = fsspec.filesystem('tar', fo='cifar-10-python.tar') for p in tfs.glob('*/data_batch_*'): batch = tfs.open(p, 'rb') batch.close() ``` `cifar-10-python.tar.gz`: ```python tfs = fsspec.filesystem('tar', fo='cifar-10-python.tar.gz', compression='gzip') for p in tfs.glob('*/data_batch_*'): batch = tfs.open(p,...
