omit chunks HTTP error 400s but not the rest

[renaudjester]

Hi!

Problem

It seems that zarr and in particular the FSStore are completely bypassing any errors coming from the server and filling with NaNs. It's good in way as it is intended for 404s and 403s but it then omits other errors that should be returned: see https://github.com/pydata/xarray/issues/8842 for the consequences of this.

Expected behavior

Maybe it should work more like the Julia library for zarr: https://github.com/JuliaIO/Zarr.jl/issues/131 where only http 400s errors are ignored. I would also expect to be able to add some retry policy to the call that fails for real (ie not the http 400s errors). Is it possible to pass a retry policy?