zarr-python icon indicating copy to clipboard operation
zarr-python copied to clipboard

Explicitly using fsspec and zarr FsspecStore causes RuntimeError "Task attached to a different loop"

Open dmitriyrepin opened this issue 3 months ago • 0 comments

Summary

Mixing explicit use of fsspec and zarr FsspecStore produces the following error: 'RuntimeError: Task <Task pending ... > attached to a different loop. Task was destroyed but it is pending!'

Reproducers:

  1. When one explicitly creates fsspec.AbstractFileSystem and then uses it in zarr.create_array(), he/she will observe error mentioned above
    gs_file = "gs://my_data/array.tmp"

    fs, url = fsspec.url_to_fs(gs_file)
    store1: fsspec.AbstractFileSystem = fs.get_mapper(url)
    arr1 = zarr.create_array(store1, shape=(2, 2), dtype='int32', overwrite=True)
    arr1[:] = np.array([[1, 2], [3, 4]], dtype='int32')
  1. When one explicitly uses fsspec functionality and then uses zarr.create_array(), he/she will observe error mentioned above:
    gs_file = "gs://my_data/array.tmp"

    fs: fsspec.AbstractFileSystem = fsspec.filesystem('gs')
    info = fs.info(gs_file)

    # Create a zarr array in the cloud
    store1: fsspec.AbstractFileSystem = zarr.storage.FsspecStore.from_url(gs_file, read_only=False)
    arr1 = zarr.create_array(store1, shape=(2, 2), dtype='int32', overwrite=True)
    arr1[:] = np.array([[1, 2], [3, 4]], dtype='int32')

Suspected cause:

When one explicitly uses fsspec, it will create asyncio thread with an event loop (the fsspecIO thread will also be created). If one later would try to use the zarr FsspecStore, which under the hood also uses fsspec, it will create the second asyncio thread with the second event loop (the zarr_io thread will also be created), which likely causes the error down the road

Image

Versions

  • zarr: 3.1.2
  • Python: 3.13.5
  • fsspec: 2025.9.0
  • gcsfs: 2025.9.0

dmitriyrepin avatar Sep 23 '25 19:09 dmitriyrepin