NVIDIA
Weather Data Download HDF Error
Following is the error I am getting while downloaded ERA5 data using start_mirror.py in examples/weather/dataset_download. I suspect the temporary nc file downloaded end up being corrupted.
Chunk for z 100 2011-1 already exists. Skipping. Downloading data for 2011-1 [# ] | 3% Completed | 230.20 s█████████▉ | 207M/242M [00:55<00:08, 4.18MB/s] Error executing job with overrides: [] c42ccd832e3e30e91a5e9f9ab3d4315c.nc: 96%|████████████████████████████████████████▏ | 232M/242M [01:00<00:01, 7.75MB/sTraceback (most recent call last): File "~/physicsnemo-fork/examples/weather/dataset_download/start_mirror.py", line 51, in main/s] zarr_paths = mirror.download(cfg.variables, date_range, hours)███▍ | 129M/197M [00:53<00:11, 6.03MB/s] File "~/physicsnemo-fork/examples/weather/dataset_download/era5_mirror.py", line 307, in download20ca1b30fdbec80d4552faace6805b3.nc: 75%|███████████████████████████████▋ | 149M/197M [00:57<00:06, 8.08MB/s] dask.compute(*tasks)c0d82f10.nc: 20%|███████▉ | 43.0M/220M [00:34<02:34, 1.20MB/s] File "~/.conda/envs/earth2mip/lib/python3.10/site-packages/dask/base.py", line 681, in computes] results = schedule(expr, keys, **kwargs) File "~/physicsnemo-fork/examples/weather/dataset_download/era5_mirror.py", line 225, in download_and_upload_chunk ds.to_zarr(mapper, mode=mode, consolidated=True, append_dim=append_dim) | 136M/173M [00:29<00:17, 2.27MB/s] File "~/.conda/envs/earth2mip/lib/python3.10/site-packages/xarray/core/dataset.py", line 2292, in to_zarrd0015ad221c726564dee25a.nc: 83%|██████████████████████████████████▉ | 144M/173M [00:34<00:15, 1.94MB/s] return to_zarr( # type: ignore[call-overload,misc] File "~/.conda/envs/earth2mip/lib/python3.10/site-packages/xarray/backends/api.py", line 2246, in to_zarr82c5ce3417469ac6467ccb6.nc: 26%|██████████▍ | 57.0M/223M [00:34<01:14, 2.33MB/s] writes = writer.sync( File "~/.conda/envs/earth2mip/lib/python3.10/site-packages/xarray/backends/common.py", line 357, in sync delayed_store = chunkmanager.store( File "~/.conda/envs/earth2mip/lib/python3.10/site-packages/xarray/namedarray/daskmanager.py", line 247, in store return store( File "~/.conda/envs/earth2mip/lib/python3.10/site-packages/xarray/core/indexing.py", line 575, in array return np.asarray(self.get_duck_array(), dtype=dtype, copy=copy) File "~/.conda/envs/earth2mip/lib/python3.10/site-packages/xarray/core/indexing.py", line 580, in get_duck_array return self.array.get_duck_array() File "~/.conda/envs/earth2mip/lib/python3.10/site-packages/xarray/core/indexing.py", line 845, in get_duck_array self.ensure_cached() File "~/.conda/envs/earth2mip/lib/python3.10/site-packages/xarray/core/indexing.py", line 842, in ensure_cached self.array = as_indexable(self.array.get_duck_array()) File "~/.conda/envs/earth2mip/lib/python3.10/site-packages/xarray/core/indexing.py", line 799, in get_duck_array return self.array.get_duck_array() File "~/.conda/envs/earth2mip/lib/python3.10/site-packages/xarray/core/indexing.py", line 654, in get_duck_array array = self.array[self.key] File "~/.conda/envs/earth2mip/lib/python3.10/site-packages/xarray/backends/netCDF4.py", line 103, in getitem return indexing.explicit_indexing_adapter( File "~/.conda/envs/earth2mip/lib/python3.10/site-packages/xarray/core/indexing.py", line 1023, in explicit_indexing_adapter result = raw_indexing_method(raw_key.tuple) File "~/.conda/envs/earth2mip/lib/python3.10/site-packages/xarray/backends/netCDF4.py", line 116, in _getitem array = getitem(original_array, key) File "src/netCDF4/_netCDF4.pyx", line 5079, in netCDF4._netCDF4.Variable.getitem File "src/netCDF4/_netCDF4.pyx", line 6051, in netCDF4._netCDF4.Variable._get File "src/netCDF4/_netCDF4.pyx", line 2164, in netCDF4._netCDF4._ensure_nc_success RuntimeError: NetCDF: HDF error
