cate
cate copied to clipboard
CCI-Tools
Failure downloading data with region constraint
@forman @HelenClifton I closed issue #783 for this new issue
Expected behavior
I expect cate to download datasets from the ODP with or without specific regions
Actual behavior
Cate was able to download data without region of constraints but returns errors for download with region of constraint
NB: CATE RETURNS 2 DIFFERENT ERRORS FOR 2 DIFFERENT DATASETS (OZONE AND LANDCOVER) BUT SAME ISSUE
e.g 1..ozone... NetCDF: Invalid argument AND e.g 2..Landcover..RuntimeError: NetCDF: Access failure
EXAMPLE 1
Sync esacci.OZONE.mon.L3.NP.multi-sensor.multi-platform.MERGED.fv0002.r1: [------------------------------] 0% ESACCI-OZONE-L3-NP-MERGED-KNMI-199701-fv0002.nc
/home/helge/.cate/data_stores/local/local.OZ_UC4_1997_2008_GHG/ESACCI-OZONE-L3-NP-MERGED-KNMI-199701-fv0002.nc
Sync esacci.OZONE.mon.L3.NP.multi-sensor.multi-platform.MERGED.fv0002.r1: [#-----------------------------] 3% ESACCI-OZONE-L3-NP-MERGED-KNMI-199702-fv0002.nc
/home/helge/.cate/data_stores/local/local.OZ_UC4_1997_2008_GHG/ESACCI-OZONE-L3-NP-MERGED-KNMI-199702-fv0002.nc
Traceback (most recent call last):
File "/home/helge/cate/cate/util/cli.py", line 266, in run_main
args_obj.command_class().execute(args_obj)
File "/home/helge/cate/cate/util/cli.py", line 163, in execute
return sub_command_function(command_args)
File "/home/helge/cate/cate/cli/main.py", line 1311, in _execute_copy
monitor=cls.new_monitor())
File "/home/helge/cate/cate/ds/esa_cci_odp.py", line 1011, in make_local
raise e
File "/home/helge/cate/cate/ds/esa_cci_odp.py", line 1004, in make_local
self._make_local(local_ds, time_range, region, var_names, monitor=monitor)
File "/home/helge/cate/cate/ds/esa_cci_odp.py", line 889, in _make_local
remote_dataset.to_netcdf(local_filepath)
File "/home/helge/miniconda3/envs/cate-env/lib/python3.6/site-packages/xarray/core/dataset.py", line 1222, in to_netcdf
compute=compute)
File "/home/helge/miniconda3/envs/cate-env/lib/python3.6/site-packages/xarray/backends/api.py", line 718, in to_netcdf
unlimited_dims=unlimited_dims)
File "/home/helge/miniconda3/envs/cate-env/lib/python3.6/site-packages/xarray/backends/api.py", line 761, in dump_to_store
unlimited_dims=unlimited_dims)
File "/home/helge/miniconda3/envs/cate-env/lib/python3.6/site-packages/xarray/backends/common.py", line 266, in store
unlimited_dims=unlimited_dims)
File "/home/helge/miniconda3/envs/cate-env/lib/python3.6/site-packages/xarray/backends/common.py", line 304, in set_variables
name, v, check, unlimited_dims=unlimited_dims)
File "/home/helge/miniconda3/envs/cate-env/lib/python3.6/site-packages/xarray/backends/netCDF4_.py", line 467, in prepare_variable
fill_value=fill_value)
File "netCDF4/_netCDF4.pyx", line 2557, in netCDF4._netCDF4.Dataset.createVariable
File "netCDF4/_netCDF4.pyx", line 3587, in netCDF4._netCDF4.Variable.__init__
File "netCDF4/_netCDF4.pyx", line 1754, in netCDF4._netCDF4._ensure_nc_success
RuntimeError: NetCDF: Invalid argument
cate ds: error: NetCDF: Invalid argument.```
EXAMPLE 2
```Traceback (most recent call last):
result = future.result()
File "C:\Users\prosper\Miniconda3\envs\cate-env\lib\concurrent\futures\_base.py", line 425, in result
return self.__get_result()
File "C:\Users\prosper\Miniconda3\envs\cate-env\lib\concurrent\futures\_base.py", line 384, in __get_result
raise self._exception
File "C:\Users\prosper\Miniconda3\envs\cate-env\lib\concurrent\futures\thread.py", line 56, in run
result = self.fn(*self.args, **self.kwargs)
File "C:\Users\prosper\Miniconda3\envs\cate-env\lib\site-packages\cate\util\web\jsonrpchandler.py", line 306, in call_service_method
result = method(*method_params, monitor=monitor)
File "C:\Users\prosper\Miniconda3\envs\cate-env\lib\site-packages\cate\webapi\websocket.py", line 292, in set_workspace_resource
monitor=monitor)
File "C:\Users\prosper\Miniconda3\envs\cate-env\lib\site-packages\cate\core\wsmanag.py", line 320, in set_workspace_resource
workspace.execute_workflow(res_name=res_name, monitor=monitor)
File "C:\Users\prosper\Miniconda3\envs\cate-env\lib\site-packages\cate\core\workspace.py", line 662, in execute_workflow
self.workflow.invoke_steps(steps, context=self._new_context(), monitor=monitor)
File "C:\Users\prosper\Miniconda3\envs\cate-env\lib\site-packages\cate\core\workflow.py", line 627, in invoke_steps
steps[0].invoke(context=context, monitor=monitor)
File "C:\Users\prosper\Miniconda3\envs\cate-env\lib\site-packages\cate\core\workflow.py", line 318, in invoke
self._invoke_impl(_new_context(context, step=self), monitor=monitor)
File "C:\Users\prosper\Miniconda3\envs\cate-env\lib\site-packages\cate\core\workflow.py", line 980, in _invoke_impl
return_value = self._op(monitor=monitor, **input_values)
File "C:\Users\prosper\Miniconda3\envs\cate-env\lib\site-packages\cate\core\op.py", line 216, in __call__
return_value = self._wrapped_op(**input_values)
File "C:\Users\prosper\Miniconda3\envs\cate-env\lib\site-packages\cate\ops\io.py", line 83, in open_dataset
monitor=monitor)
File "C:\Users\prosper\Miniconda3\envs\cate-env\lib\site-packages\cate\core\ds.py", line 588, in open_dataset
monitor=monitor.child(80))
File "C:\Users\prosper\Miniconda3\envs\cate-env\lib\site-packages\cate\ds\esa_cci_odp.py", line 1006, in make_local
raise e
File "C:\Users\prosper\Miniconda3\envs\cate-env\lib\site-packages\cate\ds\esa_cci_odp.py", line 999, in make_local
self._make_local(local_ds, time_range, region, var_names, monitor=monitor)
File "C:\Users\prosper\Miniconda3\envs\cate-env\lib\site-packages\cate\ds\esa_cci_odp.py", line 884, in _make_local
remote_dataset.to_netcdf(local_filepath)
File "C:\Users\prosper\Miniconda3\envs\cate-env\lib\site-packages\xarray\core\dataset.py", line 1150, in to_netcdf
compute=compute)
File "C:\Users\prosper\Miniconda3\envs\cate-env\lib\site-packages\xarray\backends\api.py", line 723, in to_netcdf
unlimited_dims=unlimited_dims, compute=compute)
File "C:\Users\prosper\Miniconda3\envs\cate-env\lib\site-packages\xarray\core\dataset.py", line 1075, in dump_to_store
unlimited_dims=unlimited_dims)
File "C:\Users\prosper\Miniconda3\envs\cate-env\lib\site-packages\xarray\backends\common.py", line 362, in store
variables, attributes = self.encode(variables, attributes)
File "C:\Users\prosper\Miniconda3\envs\cate-env\lib\site-packages\xarray\backends\common.py", line 447, in encode
variables, attributes = cf_encoder(variables, attributes)
File "C:\Users\prosper\Miniconda3\envs\cate-env\lib\site-packages\xarray\conventions.py", line 573, in cf_encoder
for k, v in iteritems(variables))
File "C:\Users\prosper\Miniconda3\envs\cate-env\lib\site-packages\xarray\conventions.py", line 573, in <genexpr>
for k, v in iteritems(variables))
File "C:\Users\prosper\Miniconda3\envs\cate-env\lib\site-packages\xarray\conventions.py", line 235, in encode_cf_variable
var = coder.encode(var, name=name)
File "C:\Users\prosper\Miniconda3\envs\cate-env\lib\site-packages\xarray\coding\times.py", line 384, in encode
dims, data, attrs, encoding = unpack_for_encoding(variable)
File "C:\Users\prosper\Miniconda3\envs\cate-env\lib\site-packages\xarray\coding\variables.py", line 96, in unpack_for_encoding
return var.dims, var.data, var.attrs.copy(), var.encoding.copy()
File "C:\Users\prosper\Miniconda3\envs\cate-env\lib\site-packages\xarray\core\variable.py", line 295, in data
return self.values
File "C:\Users\prosper\Miniconda3\envs\cate-env\lib\site-packages\xarray\core\variable.py", line 387, in values
return _as_array_or_item(self._data)
File "C:\Users\prosper\Miniconda3\envs\cate-env\lib\site-packages\xarray\core\variable.py", line 211, in _as_array_or_item
data = np.asarray(data)
File "C:\Users\prosper\Miniconda3\envs\cate-env\lib\site-packages\numpy\core\numeric.py", line 501, in asarray
return array(a, dtype, copy=False, order=order)
File "C:\Users\prosper\Miniconda3\envs\cate-env\lib\site-packages\xarray\core\indexing.py", line 624, in __array__
self._ensure_cached()
File "C:\Users\prosper\Miniconda3\envs\cate-env\lib\site-packages\xarray\core\indexing.py", line 621, in _ensure_cached
self.array = NumpyIndexingAdapter(np.asarray(self.array))
File "C:\Users\prosper\Miniconda3\envs\cate-env\lib\site-packages\numpy\core\numeric.py", line 501, in asarray
return array(a, dtype, copy=False, order=order)
File "C:\Users\prosper\Miniconda3\envs\cate-env\lib\site-packages\xarray\core\indexing.py", line 602, in __array__
return np.asarray(self.array, dtype=dtype)
File "C:\Users\prosper\Miniconda3\envs\cate-env\lib\site-packages\numpy\core\numeric.py", line 501, in asarray
return array(a, dtype, copy=False, order=order)
File "C:\Users\prosper\Miniconda3\envs\cate-env\lib\site-packages\xarray\core\indexing.py", line 508, in __array__
return np.asarray(array[self.key], dtype=None)
File "C:\Users\prosper\Miniconda3\envs\cate-env\lib\site-packages\xarray\coding\variables.py", line 66, in __getitem__
return self.func(self.array[key])
File "C:\Users\prosper\Miniconda3\envs\cate-env\lib\site-packages\xarray\coding\variables.py", line 66, in __getitem__
return self.func(self.array[key])
File "C:\Users\prosper\Miniconda3\envs\cate-env\lib\site-packages\numpy\core\numeric.py", line 501, in asarray
return array(a, dtype, copy=False, order=order)
File "C:\Users\prosper\Miniconda3\envs\cate-env\lib\site-packages\xarray\core\indexing.py", line 508, in __array__
return np.asarray(array[self.key], dtype=None)
File "C:\Users\prosper\Miniconda3\envs\cate-env\lib\site-packages\xarray\backends\netCDF4_.py", line 66, in __getitem__
array = getitem(self.get_array(), key.tuple)
File "C:\Users\prosper\Miniconda3\envs\cate-env\lib\site-packages\xarray\backends\common.py", line 116, in robust_getitem
return array[key]
File "netCDF4\_netCDF4.pyx", line 4060, in netCDF4._netCDF4.Variable.__getitem__
File "netCDF4\_netCDF4.pyx", line 4950, in netCDF4._netCDF4.Variable._get
File "netCDF4\_netCDF4.pyx", line 1733, in netCDF4._netCDF4._ensure_nc_success
RuntimeError: NetCDF: Access failure*```
### Steps to reproduce the problem
1. download ozone and landcover from ODP without specifying values for region of constraint
2. repeat the same process but provide values for regions to be delimited
### Specifications
Cate 2.0.0-dev 23, windows 10
@forman I got those error messages when I specified the region constraints for both the Ozone L3 merged and Landcover as lon -20, 20 lat -20, 20. I tried today for both data but the Ozone was successful but that of the landcover failed(same as #783). Should I close this issue and reopen #783?