Peter Marsh

Hi @aaronspring @rsignell-usgs, Here is an example reproducing a dataset similar to that of climpreds using kerchunk: https://nbviewer.org/gist/peterm790/1ad0194df364b6efd51631ecaa96deda The structure of it isn't ideal with all the dask warnings, but...

Hi @rbanick, It looks like the Kerchunk package you are using is a bit older as the bug from `o = selector.match(fn).groups[0]` has already been fixed by @martindurant in https://github.com/fsspec/kerchunk/commit/5cfa8871d86fe165213cc18fef042cf82258a573...

Hi @martindurant, Thank you very much for accepting my proposal and welcoming me to the community I look forward working with you all!

Hello @rsignell-usgs, yes 1pm UTC tomorrow will work for me, I will reach out via email.

Yes, I have sent a second from an alternate email incase the previous did not make it past a spam filter. My preferred email is [email protected]

So I have had a go at testing the two methods suggested and both result in a speed up: | Normal MZZ | Tree Reduction | With `fs.cat` | Tree...

> How many cores did dask use? If 4, then that's really pretty good parallelism. This was 8 cores

> You would use `fsspec.core.url_to_fs` (be sure to pass **target_options) Thanks, I have updated the branch to use this: https://github.com/fsspec/kerchunk/compare/main...peterm790:kerchunk:fs_cat_combine

Aaah sorry I was under the impression `cramjam` alone would be enough to handle the decompression. In that case this seems reasonable

Right! I suppose modifying: https://github.com/fsspec/filesystem_spec/blob/2633445fc54797c79a7ac96d213bd24dfbdfcdc2/fsspec/core.py#L487 to rather raise: "Compression support for type %s not found" or even "Install X to access compression support for %s" Might make the fact that...