filesystem_spec
filesystem_spec copied to clipboard
fsspec
A specification that python filesystems should adhere to.
I noticed that fsspec has `fsspec.compression` but it's not documented anywhere how to use it. Even a simple example would help on how to create a fs using `fsspec` and...
https://github.com/fsspec/filesystem_spec/blob/master/fsspec/asyn.py#L27-L35 If `_lock` is None, if two threads both enter `get_lock`, both threads could see `_lock` is None, one thread could create a new `Lock`, return it to the caller,...
Here's a minimal example: ```python import fsspec import asyncio async def async_version(): print("Async Version") fs = fsspec.filesystem("http", asynchronous=True) session = await fs.set_session() file = await fs.open_async("https://example.com/") print("Starting Tell", file.tell(), "seeking...
this snippet takes ~24s to run on my laptop in Germany: ```python # /// script # requires-python = ">=3.11" # dependencies = [ # "fsspec", # "gcsfs", # ] #...
I've run a [test build with Zarr 3](https://copr.fedorainfracloud.org/coprs/qulogic/zarr3/build/8547554/) This update has caused some tests to fail due to some issue in `LazyReferenceMapper`: ```pytb ____________________ ERROR at setup of test_append_parquet _____________________...
The `fsspec.open` command allows `urlpath` to be a list. But [a few lines later](https://github.com/fsspec/filesystem_spec/blob/30af5e1d1f201d8681faf0ca163c4c9509de69a3/fsspec/core.py#L492), urlpath is passed as a list. If it is already a list, it becomes a list...
See https://github.com/fsspec/filesystem_spec/issues/625#issuecomment-2332915497
The problem is that when `HTTPFileSystem.get is called`, it checks if it is a directory [here](https://github.com/fsspec/filesystem_spec/blob/90c7cd9e6c939fc37341fd793831a399753ebfd9/fsspec/asyn.py#L639), but to check that this is a directory, it downloads the whole body of...
I'm trying to implement a file system that always replaces the `'a'` open mode with `'w'`. The purpose is to support TensorBoard on filesystems (local mounts of remote network FS,...
Hi! Just got a feature idea that might be a nice-to-have maybe?: supporting taking `.ssh/config` for parameters i.e. just like we do `sftp nickname` in console, maybe we could connect...
