`auto_mkdir` not propagated resulting in FileNotFoundError during `write_image`

[npeschke]

I am trying to save a multidimensional image using ome-zarr, however saving fails with the following error:

FileNotFoundError: [Errno 2] No such file or directory: '/path/to/input_image.ome.zarr/0/0/0/0'

I am using the following code to save the image:

loc = parse_url(path, mode="w").store
group = zarr.group(store = loc)

n_channels = input_image.shape[0]
channels = [f"Channel{i}" for i in range(n_channels)]

group.attrs["omero"] = {
    "name": self.DEFAULT_INPUT_IMAGE_NAME,
    "channels": [{"label":channel, "color":self.channel_colors[i], "active":True} for i, channel in enumerate(channels)]
}

write_image(input_image, group = group, axes = "cyx", storage_options=dict(chunks=(1, 1024, 1024)))

The following directory structure is generated:

input_image.ome.zarr/
├── 0
│   └── .zarray
├── .zattrs
└── .zgroup

I've narrowed down the issue to auto_mkdir not being propagated to the LocalFileSystem object in the store

Is this intended behaviour?

[will-moore]

Hi, thanks for the feedback. I was just trying to reproduce this issue locally, and failing until I was notified of https://github.com/ome/ome-zarr-py/issues/302, which I believe is the same issue.

On updating my local version of fsspec-2022.2.0 to fsspec-2023.9.0 also (s3fs-2022.2.0 to s3fs-2023.9.0) I now see the same issue. So it looks like the short-term solution is to install fsspec-2022.2.0 (assuming you have a more recent version) while we try to fix the issue.

[will-moore]

Seems the commit at https://github.com/fsspec/filesystem_spec/commit/285094f0771dcc98467bf0a0e5e5b14a6cf096b3 is involved in this issue: https://github.com/fsspec/filesystem_spec/commit/285094f0771dcc98467bf0a0e5e5b14a6cf096b3

[will-moore]

Same issue reported at #305 (with stack trace) and also causes test failures at #302.

See discussion on this issue at https://github.com/fsspec/filesystem_spec/commit/285094f0771dcc98467bf0a0e5e5b14a6cf096b3

[davidpross]

I got the same issue running the example here. I had fsspec 2023.9.0, running pip install fsspec=="2022.2.0" resolved the issue. Thanks.