filesystem_spec

filesystem_spec copied to clipboard

I'm packaging your module as an rpm package so I'm using the typical PEP517 based build, install and test cycle used on building packages from non-root account. - `python3 -sBm...

kloczek

HTTPFilesystem has a race condition on data size between the open and read calls, if content changes at server between the 2 calls

5

The following script reproduces the issues The script spins up an http server that makes the json content 1 chat longer every 1s. Then the client bit hits the url...

masariello

add fastparquet dependency option

1

This is necessary for getting tests to work with `pip install .[full]`

bendichter

allow ReferenceFileSystem to hold dicts, which are treated as JSON files

4

Currently, when a ReferenceFileSystem wants to create an inline JSON file, the value needs to be a JSON string, e.g. ```json { "version": 1, "refs": { ".zgroup": "{\"zarr_format\": 2}", "data/.zarray":...

bendichter

use pyarrow in LazyReferenceMapper.write?

2

Currently the `LazyReferenceMapper.write` uses fastparquet to write the parquet file (https://github.com/fsspec/filesystem_spec/blob/master/fsspec/implementations/reference.py#L467) I came across this as I didn't have fastparquet in my env. It would be nice to have a...

raybellwaves

Closes #1563 Not tested.

raybellwaves

Fixes #1541: HTTPFilesystem has a race condition on data size between the open and read class, if content changes at server between the 2 class

2

masariello

S3 URL containing versionId is misinterpreted as glob pattern

8

I am trying to load a specific version of an S3 object into a Dask dataframe as follows: ```python >>> import dask.dataframe as dd >>> dd.read_parquet("s3://example-bucket/dataset/part.0.parquet?versionId=se654wJoRQcirhKoHRkN4hsFmhNwMg27") ``` This example returns...

mdwint

Add ctime/mtime to list of expected values in info

5

Created and/or modified time is returned in the file info of most backends. We should endeavour to surface these in the file info dict with a common format (`datetime.datetime`? unix...

martindurant

FSSpec's get_mapper and other functions can produce nameless files for S3 Filesystems

6

### Observed Behavior: When `fsspec`'s S3FS's `get_mapper` function is called on an empty directory-like path in an S3 filesystem, it can produce a dictionary-like object that is non-empty, and includes...

jaedoucette