Reference Filesystem: "ValueError: Compression type zstd not supported"

[peterm790]

When opening a zstd compressed reference file using:

fs = fsspec.filesystem("reference", fo='s3://esip-qhub-public/ecmwf/ERA5_2020_2022_multivar.json.zst', ref_storage_args= {"compression": "zstd"}, remote_protocol='s3', remote_options={'anon':True})

I get the error:

Traceback (most recent call last):
  File "/home/peter/Documents/intake_test/test_local.py", line 3, in <module>
    fs = fsspec.filesystem("reference", fo='ERA5_2020_2022_multivar.json.zst',ref_storage_args={"compression": "zstd"})
  File "/home/peter/anaconda3/envs/open_intake_min/lib/python3.10/site-packages/fsspec/registry.py", line 262, in filesystem
    return cls(**storage_options)
  File "/home/peter/anaconda3/envs/open_intake_min/lib/python3.10/site-packages/fsspec/spec.py", line 76, in __call__
    obj = super().__call__(*args, **kwargs)
  File "/home/peter/anaconda3/envs/open_intake_min/lib/python3.10/site-packages/fsspec/implementations/reference.py", line 140, in __init__
    with open(fo, "rb", **dic) as f:
  File "/home/peter/anaconda3/envs/open_intake_min/lib/python3.10/site-packages/fsspec/core.py", line 467, in open
    return open_files(
  File "/home/peter/anaconda3/envs/open_intake_min/lib/python3.10/site-packages/fsspec/core.py", line 312, in open_files
    [
  File "/home/peter/anaconda3/envs/open_intake_min/lib/python3.10/site-packages/fsspec/core.py", line 313, in <listcomp>
    OpenFile(
  File "/home/peter/anaconda3/envs/open_intake_min/lib/python3.10/site-packages/fsspec/core.py", line 74, in __init__
    self.compression = get_compression(path, compression)
  File "/home/peter/anaconda3/envs/open_intake_min/lib/python3.10/site-packages/fsspec/core.py", line 513, in get_compression
    raise ValueError("Compression type %s not supported" % compression)
ValueError: Compression type zstd not supported

This occurs in a clean environment with the latest fsspec and s3fs installed. Running pip install zstandard fixes the error.

[martindurant]

This is expected, fsspec has no dependencies of its own. Do you think the message should be cleaner? See also #990

[peterm790]

Aaah sorry I was under the impression cramjam alone would be enough to handle the decompression. In that case this seems reasonable

[martindurant]

cramjam would be enough, but the PR was not merged as we decided that cramjam should not be a requirement either. It should be finished off such that cramjam alone would be enough to cover all the compression codecs, but maybe still allow for the current set of optional dependencies as fallback.

[peterm790]

Right! I suppose modifying:

https://github.com/fsspec/filesystem_spec/blob/2633445fc54797c79a7ac96d213bd24dfbdfcdc2/fsspec/core.py#L487

to rather raise:

"Compression support for type %s not found" or even "Install X to access compression support for %s"

Might make the fact that zstd is supported just not with the current environment a bit clearer.