Duplicated layers loaded with s3fs & fsspec 2023.12.2

[jluethi]

I'm running into an issue when setting up fresh environments with the napari-ome-zarr plugin and could trace it back to an interaction with fsspec==2023.12.2 & s3fs==2023.12.2.

Using different OME-Zarr datasets (for example this public dataset: https://zenodo.org/records/10257149), I see the same layers being loaded multiple times. Sometimes a layer is loaded 3x, sometimes 46x. The same dataset has different behaviors whether I load it from my local SSD (more duplicates) or slower remote storage.

The issue is reproducible for local storage and remote shares, on Windows & Mac, with napari 0.4.18, 0.4.19 & 0.5.0dev versions from the main branch.

The thing that fixes it is reverting to an older version of fsspec & s3fs, e.g. by doing: pip install s3fs==2023.9.2

---

I couldn't trace the underlying issue yet. I don't know if it's a bug in fsspec, s3fs or the way ome-zarr py is using the libraries. Just thought I'd document it here for the moment in case other napari-ome-zarr users come across it. Downgrading the fsspec & s3fs versions does solve it (that was a wasted afternoon to figure out, so maybe it saves someone else time to find this here ;))