cate icon indicating copy to clipboard operation
cate copied to clipboard
verified publisher icon CCI-Tools

Fire: Copying remote data failed

Open Evadzi opened this issue 7 years ago • 0 comments

Expected behavior

Fire data ingestion from ODP

Actual behavior

Failed to copy remote data

Steps to reproduce the problem

  1. download fire data using ODP and set region as lon -20, 20 lat -20, 20

….returns error:-

Message: Data source "esacci.FIRE.day.L4.BA.multi-sensor.multi-platform.MERIS.v4-1.r1": Copying remote data source failed: [Errno -68] NetCDF: I/O failure: b'http://esgf-data1.ceda.ac.uk/thredds/dodsC/esg_esacci/fire/data/burned_area/grid/v4.1/2009/20090207-ESACCI-L4_FIRE-BA-MERIS-fv4.1.nc'
Code: -32003
Method: set_workspace_resource
Exception: cate.core.ds.DataAccessError

Traceback (most recent call last):
  File "C:\Users\prosper\cate-2.0.0.dev22\lib\site-packages\cate\ds\esa_cci_odp.py", line 874, in _make_local
    remote_dataset = adjust_spatial_attrs_impl(subset_spatial_impl(remote_dataset, region),
  File "C:\Users\prosper\cate-2.0.0.dev22\lib\site-packages\cate\core\opimpl.py", line 880, in subset_spatial_impl
    return reset_non_spatial(ds, retset)
  File "C:\Users\prosper\cate-2.0.0.dev22\lib\site-packages\cate\core\opimpl.py", line 782, in reset_non_spatial
    for var_name in ds_source.var():
  File "C:\Users\prosper\cate-2.0.0.dev22\lib\site-packages\xarray\core\common.py", line 54, in wrapped_func
    **kwargs)
  File "C:\Users\prosper\cate-2.0.0.dev22\lib\site-packages\xarray\core\dataset.py", line 2635, in reduce
    **kwargs)
  File "C:\Users\prosper\cate-2.0.0.dev22\lib\site-packages\xarray\core\variable.py", line 1263, in reduce
    data = func(self.data if allow_lazy else self.values,
  File "C:\Users\prosper\cate-2.0.0.dev22\lib\site-packages\xarray\core\variable.py", line 295, in data
    return self.values
  File "C:\Users\prosper\cate-2.0.0.dev22\lib\site-packages\xarray\core\variable.py", line 387, in values
    return _as_array_or_item(self._data)
  File "C:\Users\prosper\cate-2.0.0.dev22\lib\site-packages\xarray\core\variable.py", line 211, in _as_array_or_item
    data = np.asarray(data)
  File "C:\Users\prosper\cate-2.0.0.dev22\lib\site-packages\numpy\core\numeric.py", line 501, in asarray
    return array(a, dtype, copy=False, order=order)
  File "C:\Users\prosper\cate-2.0.0.dev22\lib\site-packages\xarray\core\indexing.py", line 624, in __array__
    self._ensure_cached()
  File "C:\Users\prosper\cate-2.0.0.dev22\lib\site-packages\xarray\core\indexing.py", line 621, in _ensure_cached
    self.array = NumpyIndexingAdapter(np.asarray(self.array))
  File "C:\Users\prosper\cate-2.0.0.dev22\lib\site-packages\numpy\core\numeric.py", line 501, in asarray
    return array(a, dtype, copy=False, order=order)
  File "C:\Users\prosper\cate-2.0.0.dev22\lib\site-packages\xarray\core\indexing.py", line 602, in __array__
    return np.asarray(self.array, dtype=dtype)
  File "C:\Users\prosper\cate-2.0.0.dev22\lib\site-packages\numpy\core\numeric.py", line 501, in asarray
    return array(a, dtype, copy=False, order=order)
  File "C:\Users\prosper\cate-2.0.0.dev22\lib\site-packages\xarray\core\indexing.py", line 508, in __array__
    return np.asarray(array[self.key], dtype=None)
  File "C:\Users\prosper\cate-2.0.0.dev22\lib\site-packages\xarray\backends\netCDF4_.py", line 64, in __getitem__
    with self.datastore.ensure_open(autoclose=True):
  File "C:\Users\prosper\cate-2.0.0.dev22\lib\contextlib.py", line 81, in __enter__
    return next(self.gen)
  File "C:\Users\prosper\cate-2.0.0.dev22\lib\site-packages\xarray\backends\common.py", line 497, in ensure_open
    self._ds = self._opener()
  File "C:\Users\prosper\cate-2.0.0.dev22\lib\site-packages\xarray\backends\netCDF4_.py", line 231, in _open_netcdf4_group
    ds = nc4.Dataset(filename, mode=mode, **kwargs)
  File "netCDF4\_netCDF4.pyx", line 2111, in netCDF4._netCDF4.Dataset.__init__
  File "netCDF4\_netCDF4.pyx", line 1731, in netCDF4._netCDF4._ensure_nc_success
OSError: [Errno -68] NetCDF: I/O failure: b'http://esgf-data1.ceda.ac.uk/thredds/dodsC/esg_esacci/fire/data/burned_area/grid/v4.1/2009/20090207-ESACCI-L4_FIRE-BA-MERIS-fv4.1.nc'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:\Users\prosper\cate-2.0.0.dev22\lib\site-packages\cate\util\web\jsonrpchandler.py", line 209, in send_service_method_result
    result = future.result()
  File "C:\Users\prosper\cate-2.0.0.dev22\lib\concurrent\futures\_base.py", line 425, in result
    return self.__get_result()
  File "C:\Users\prosper\cate-2.0.0.dev22\lib\concurrent\futures\_base.py", line 384, in __get_result
    raise self._exception
  File "C:\Users\prosper\cate-2.0.0.dev22\lib\concurrent\futures\thread.py", line 56, in run
    result = self.fn(*self.args, **self.kwargs)
  File "C:\Users\prosper\cate-2.0.0.dev22\lib\site-packages\cate\util\web\jsonrpchandler.py", line 306, in call_service_method
    result = method(*method_params, monitor=monitor)
  File "C:\Users\prosper\cate-2.0.0.dev22\lib\site-packages\cate\webapi\websocket.py", line 292, in set_workspace_resource
    monitor=monitor)
  File "C:\Users\prosper\cate-2.0.0.dev22\lib\site-packages\cate\core\wsmanag.py", line 320, in set_workspace_resource
    workspace.execute_workflow(res_name=res_name, monitor=monitor)
  File "C:\Users\prosper\cate-2.0.0.dev22\lib\site-packages\cate\core\workspace.py", line 662, in execute_workflow
    self.workflow.invoke_steps(steps, context=self._new_context(), monitor=monitor)
  File "C:\Users\prosper\cate-2.0.0.dev22\lib\site-packages\cate\core\workflow.py", line 627, in invoke_steps
    steps[0].invoke(context=context, monitor=monitor)
  File "C:\Users\prosper\cate-2.0.0.dev22\lib\site-packages\cate\core\workflow.py", line 318, in invoke
    self._invoke_impl(_new_context(context, step=self), monitor=monitor)
  File "C:\Users\prosper\cate-2.0.0.dev22\lib\site-packages\cate\core\workflow.py", line 980, in _invoke_impl
    return_value = self._op(monitor=monitor, **input_values)
  File "C:\Users\prosper\cate-2.0.0.dev22\lib\site-packages\cate\core\op.py", line 216, in __call__
    return_value = self._wrapped_op(**input_values)
  File "C:\Users\prosper\cate-2.0.0.dev22\lib\site-packages\cate\ops\io.py", line 83, in open_dataset
    monitor=monitor)
  File "C:\Users\prosper\cate-2.0.0.dev22\lib\site-packages\cate\core\ds.py", line 588, in open_dataset
    monitor=monitor.child(80))
  File "C:\Users\prosper\cate-2.0.0.dev22\lib\site-packages\cate\ds\esa_cci_odp.py", line 1010, in make_local
    raise e
  File "C:\Users\prosper\cate-2.0.0.dev22\lib\site-packages\cate\ds\esa_cci_odp.py", line 1003, in make_local
    self._make_local(local_ds, time_range, region, var_names, monitor=monitor)
  File "C:\Users\prosper\cate-2.0.0.dev22\lib\site-packages\cate\ds\esa_cci_odp.py", line 947, in _make_local
    raise DataAccessError("Copying remote data source failed: {}".format(e), source=self) from e
cate.core.ds.DataAccessError: Data source "esacci.FIRE.day.L4.BA.multi-sensor.multi-platform.MERIS.v4-1.r1": Copying remote data source failed: [Errno -68] NetCDF: I/O failure: b'http://esgf-data1.ceda.ac.uk/thredds/dodsC/esg_esacci/fire/data/burned_area/grid/v4.1/2009/20090207-ESACCI-L4_FIRE-BA-MERIS-fv4.1.nc'``

### Specifications 

cate 2.0.0-dev 23, windows 10

Evadzi avatar Nov 29 '18 15:11 Evadzi