filesystem_spec

filesystem_spec copied to clipboard

Add url spec to filesystem

2

For some file providers it's handy to get a url for a specfici file. For example for S3 you may want a pre-signed url you can give to another server...

jcampbell05

concurrent.futures.ProcessPoolExecutor fails for HTTPFileSystem

5

@CodyCBakerPhD and I have been exploring different ways of parallelizing with http fsspec. We have found that you can use multiple processes in joblib (1), and parallelizing over threads using...

bendichter

IndexError: list index out of range when 400 HTTP errors

12

If we get a 400 HTTP error while trying to open a URL with: ```python fsspec.open(url) ``` this is not caught and instead an `IndexError` is raised: ``` IndexError: list...

albertvillanova

bugfix: await on size, assuming it can be an async function

6

Checking `size` may require a call to the async `_info` function. On line 277, `_cp_file` checks if `f1.size` is async (`callback.set_size(await maybe_await(f1.size))`), but then later it simply references `f1.size` which...

mukhery

xref: https://github.com/fsspec/community/issues/8

martindurant

Comparing an AbstractBufferedFile (or subclass) with some objects might raise an exception

2

The [`__eq__` method of `AbstractBufferedFile`](https://github.com/fsspec/filesystem_spec/blob/95eb5f94d3230fcfa95052eef6cad1d951431ff5/fsspec/spec.py#L1553) assumes that the `other` element of the comparison has the `.mode` attribute: https://github.com/fsspec/filesystem_spec/blob/95eb5f94d3230fcfa95052eef6cad1d951431ff5/fsspec/spec.py#L1553 However, this is not always the case. E.g. one could decide to...

giovannipizzi

Can't access data from S3 Buckets

14

``` import fsspec s3_fs = fsspec.filesystem("s3", key="xxxxxx', secret="xxxxxxxxx", client_kwargs={ "region_name": 'xxxxxxx', }) s3_fs.ls('s3://') ``` It lists all the buckets but when i use `s3_fs.ls('s3://Bucket_name') ` It returns empty Same bucket...

omertechverx

Glue 1.0 pythonshell job with Python 3.6 with awswrangler is failing with - "ModuleNotFoundError: No module named 'importlib.metadata'"

5

awswrnagler installed latest version for fsspec -  error - 

rsingh-821

Inconsistencies: `ls` may have `detail=True` or `detail=False`

1

I saw the `AbstractArchiveFileSystem` switched to `detail=True` in https://github.com/fsspec/filesystem_spec/commit/0f3ecd8e629043646ab19b1a2b00d895f0553a81. There are some inconstencies with other filesystems which have `detail=False` (e.g. the local filesystem). with `fsspec==2023.3.0` Currently: ```python >>> import fsspec...

lhoestq

Release process

5

I was expecting there to be a February release but looks like there wasn't one. I am curious about the release cadence of fsspec, I don't see a document in...

f4hy