Tolerate 403 HTTP codes in S3 stores as "non existing" objects

[jordi-domingo]

Dear all,

I have a request for Zarr-Python related to their S3 stores (fsspec, obstore).

Under certain circumstances, accessing a Zarr datacube in S3 can lead to 403 responses, which is an expected and correct behaviour:

1. During the creation of the datacube, chunk files with no data are directly not created (instead of left "empty").
2. S3 access is restricted (with or without credentials), and user does no have ListObjects permissions.

For these cases, S3 returns 403 instead of 404 (which is correct). However, Zarr-Python fails with PermissionError.

When the user credentials have ListObjects permissions and S3 returns 404, Zarr-Python tolerates fine these non-exisitng chunks and the call (open_zarr or whatever) works well.

I suggest tolerating 403 responses in S3 stores in the same way 404 are.

Please, forgive me if this is not the right place to request this and it belong more to fsspec or obstore repos.

[d-v-b]

hi @jordi-domingo I think we support this today by including the 403 exception class in the allowed_exceptions keyword argument for the fsspec store constructor. When the fsspec store encounters one of those exceptions while performing a get, it treats the result as None (i.e., the same as a missing object)

there's probably ways we could make this API better!

[jordi-domingo]

hi @jordi-domingo I think we support this today by including the 403 exception class in the allowed_exceptions keyword argument for the fsspec store constructor. When the fsspec store encounters one of those exceptions while performing a get, it treats the result as None (i.e., the same as a missing object)

there's probably ways we could make this API better!

Dear @d-v-b ,

Thanks, it works! For now I'll use it this way.

Still, I suggest adding a more user-friendly way of doing it; or adding PermissionError as one of the default values in ALLOWED_EXCEPTIONS.

[d-v-b]

I totally agree that this method is not user friendly! A basic problem here is that if we treat all permission errors the same, we might hide real permission errors from users. Ultimately I think we need to fix in an abstraction layer above the store classes.

[jordi-domingo]

Btw, I could make it work with FsspecStore. However, for obstore I don't see a way of passing the allowed_exceptions. Maybe it is something that can be added?