intake-esm
intake-esm copied to clipboard
intake
Fix tests
Change Summary
Pin netCDF4 to a version lower than 1.6. I've lost track of the issues talking about the problems of netCDF4 1.6.1, but downgrading solved the errors in all CI errors, but one. I'm not sure, but this could be the culprit issue : https://github.com/Unidata/netcdf4-python/issues/1192
The test_source.py::test_open_dataset_kerchunk test is failing in a weird way. It seems that at some point, fsspec fails with a ConnectionTimeoutError, however, this error is not raised. Instead somehow replaces an object and the fsspec later tries to access an item of that object, this time raising TypeError: 'ConnectTimeoutError' object is not subscriptable.
The noaa-nwm-test-reference.json file made reference to the S3 bucket noaa-nwm-retro-v2.0-pds and folder full_physics. I have not found this bucket by googling it. When trying to access the http endpoint of s3 directly, it failed. However, I did find this:
https://registry.opendata.aws/nwm-archive/ . Was the old bucket simply renamed and reorganized? This was done in #480.
@andersy005 any ideas?
I replaced all reference of the old urls with a new one that imitates the one that works for me in the http browser, but it still fails.
Checklist
- [ ] Unit tests for the changes exist
- [ ] Tests pass on CI
- [ ] Documentation reflects the changes where applicable
Oh! I didn't realize the error was different with the changed s3 paths. Now it's a "NoCredentialsError". Was the previous bucket public and the new one isn't? I'm afraid I don't know enough about AWS to solve this.
@aulemahal, @mgrover1 fixed the kerchunk issue in
- https://github.com/intake/intake-esm/pull/548
However, the netCDF's thread safety issue is still outstanding.
@andersy005 - it appears to be netcdf-c issue
https://github.com/Unidata/netcdf4-python/issues/1192#issuecomment-1273637014
