filesystem_spec
filesystem_spec copied to clipboard
fsspec
A specification that python filesystems should adhere to.
I didn't find an existing issue, so I am opening a new one. It looks like we can't pass pathlib Path's as local paths for get. I've only tried it...
Does fsspec provide any API to assemble urlpath or local path from given fs or protocol? e.g ```# Any API like def get_path(protocol, file_path): # decide which pattern to use...
Dear @martindurant and fsspec team, this PR adds an additional file-based listings cache with the help of the beautiful diskcache[1]. This is relevant as the listings cache docs mention that...
See discussion on https://stackoverflow.com/questions/70658956/paramiko-sftp-file-renaming-oserrorextended-request-not-supported/70668235. The posix_rename() appears to limit the usage of the SFTPFileSystem.rename() method to sFTP servers that run on Posix-like servers.
This may be related to https://github.com/fsspec/filesystem_spec/issues/517, but the workaround suggested there does not seem to work... ```python # From https://github.com/fsspec/filesystem_spec/issues/517 # Adapted for s3fs import multiprocessing import time import pandas...
We have a github implementation, which I find surprisingly convenient. We should also do one for gists, which also have a reasonable API https://docs.github.com/en/rest/reference/gists . We could mirror the github...
## Background The recently-added [`open_parquet_file`](https://github.com/fsspec/filesystem_spec/blob/76da18cf5a9697f480e5a0f6d1013d71676af131/fsspec/parquet.py#L17) function introduces a new variation on the existing `AbstractBufferedFile` caching strategies. That is, it compliments the `KnownPartsOfAFile` ("parts") strategy by performing format-aware "precaching" before the...
The behavior of put is not consistent with the linux command ```cp```. When we have source and target paths like source : "path1/file1.txt" target : "path2/" and Target directory already...
When failing to access a http url (e.g., a network error), `FileNotFoundError` will be raised here: https://github.com/fsspec/filesystem_spec/blob/bdd752544ac4629deb4565103e4e42855af277a8/fsspec/implementations/http.py#L393-L397 Should this be considered a bug since `FileNotFoundError` is usually be used to...
Currently, `ReferenceFileSystem` takes an argument `fo`: https://github.com/fsspec/filesystem_spec/blob/6233f315548b512ec379323f762b70764efeb92c/fsspec/implementations/reference.py#L66-L69 It's not clear to me what `fo` means. I understand that is the dictionary generated by kerchunk. Does that have a concrete name...
