filesystem_spec

filesystem_spec copied to clipboard

Just as fastparquet found, having all the compressors in the fast and compact cramjam library really makes things simple. I am suggesting that, if cramjam is installed, we should get...

martindurant

DirFileSystem.ls(path, detail=True) hits assertion error with Azure Blob Storage

12

I ran into an error using DirFileSystem with Azure Blob Storage. Here's the minimum code to expose this error: ```python import fsspec from fsspec.implementations.dirfs import DirFileSystem bucket = "my-bucket" fs...

benlindsay

docs: text says S3, but example is for GCS

1

Text: "For example to be able to access data in S3, you can use the optional pip install syntax below, or install the specific package required" but then the pip...

philvarner

Defining `timeout` as a YAML serializable object as part of fsspec storage_options instead of an `aiohttp.ClientTimeout` object

4

Currently, the only way (as far as I know) to specify `timeout` as part of `storage_options` requires instantiating an `aiohttp.ClientTimeout` object ```python In [46]: import aiohttp, fsspec In [49]: timeout...

andersy005

Entrypoint support for abfss in adlfs prevent this protocol to be overwritten by another package

3

Hi! Sorry if this issue is misplaced and should go to the adlfs package instead. I already created an issue there, but after thinking of it, maybe it is more...

pierre-godard

Standardize content_type handling when writing files

5

Would it be helpful to standardize the ability to set the content type of files written by fsspec? Several backends have / are requesting the ability to set this (https://github.com/fsspec/s3fs/blob/4f289eaa34dfe8337a72f5a0148c41a44793fde0/s3fs/core.py#L971-L974,...

TomAugspurger

read an HDFS file by line failed when the open_mode is "rb"

7

Hi there, I found some problems when I read an HDFS file by line when the open_mode is "rb". It works fine when the open_mode is "r" or the file...

amznero

DirFileSystem.ls() has `detail=True` as default

7

In the DirFileSystem implementation, the `detail` parameter is set to `True` by default (see [here](https://github.com/fsspec/filesystem_spec/blob/master/fsspec/implementations/dirfs.py#L209) and [here](https://github.com/fsspec/filesystem_spec/blob/master/fsspec/implementations/dirfs.py#L200)) whereas for other implementations like LocalFileSystem, it is set to `False` (see [here](https://github.com/fsspec/filesystem_spec/blob/master/fsspec/implementations/local.py#L53))....

benlindsay

Recursive directory copy between different filesystem

2

Is there is a way to recursively copy a directory between one file system and another? It seems like `put` is only restricted to `local -> fs` but not `fs...

hadim

issues with default block_size

6

Thanks for the wonderful library! I have an issue using data from a remote http url, here is a example snippet ( originally, the issue arose while using the dataset...

suvarchal