cate icon indicating copy to clipboard operation
cate copied to clipboard
verified publisher icon CCI-Tools

Failed to download Sea-Ice CCI data: ValueError

Open Evadzi opened this issue 7 years ago • 9 comments

Expected behavior

Download Sea-Ice data from ODP

Actual behavior

Failed

Steps to reproduce the problem

  1. select the sea-ice product = esacci.SEAICE.day.L4.SICONC.multi-sensor.multi-platform.AMSR_25kmEASE2.2-1.NH
  2. Time Constraint: 2002-08-01,2002-09-30 3.Region constraint: Lon -40 40, Lat -50 80 error report...…
Message: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
Code: -32003
Method: set_workspace_resource
Exception: ValueError

Traceback (most recent call last):
  File "c:\users\prosper\miniconda3\envs\cate-env\lib\site-packages\cate-2.0.0.dev24-py3.6.egg\cate\util\web\jsonrpchandler.py", line 209, in send_service_method_result
    result = future.result()
  File "C:\Users\prosper\Miniconda3\envs\cate-env\lib\concurrent\futures\_base.py", line 425, in result
    return self.__get_result()
  File "C:\Users\prosper\Miniconda3\envs\cate-env\lib\concurrent\futures\_base.py", line 384, in __get_result
    raise self._exception
  File "C:\Users\prosper\Miniconda3\envs\cate-env\lib\concurrent\futures\thread.py", line 56, in run
    result = self.fn(*self.args, **self.kwargs)
  File "c:\users\prosper\miniconda3\envs\cate-env\lib\site-packages\cate-2.0.0.dev24-py3.6.egg\cate\util\web\jsonrpchandler.py", line 306, in call_service_method
    result = method(*method_params, monitor=monitor)
  File "c:\users\prosper\miniconda3\envs\cate-env\lib\site-packages\cate-2.0.0.dev24-py3.6.egg\cate\webapi\websocket.py", line 292, in set_workspace_resource
    monitor=monitor)
  File "c:\users\prosper\miniconda3\envs\cate-env\lib\site-packages\cate-2.0.0.dev24-py3.6.egg\cate\core\wsmanag.py", line 320, in set_workspace_resource
    workspace.execute_workflow(res_name=res_name, monitor=monitor)
  File "c:\users\prosper\miniconda3\envs\cate-env\lib\site-packages\cate-2.0.0.dev24-py3.6.egg\cate\core\workspace.py", line 662, in execute_workflow
    self.workflow.invoke_steps(steps, context=self._new_context(), monitor=monitor)
  File "c:\users\prosper\miniconda3\envs\cate-env\lib\site-packages\cate-2.0.0.dev24-py3.6.egg\cate\core\workflow.py", line 627, in invoke_steps
    steps[0].invoke(context=context, monitor=monitor)
  File "c:\users\prosper\miniconda3\envs\cate-env\lib\site-packages\cate-2.0.0.dev24-py3.6.egg\cate\core\workflow.py", line 318, in invoke
    self._invoke_impl(_new_context(context, step=self), monitor=monitor)
  File "c:\users\prosper\miniconda3\envs\cate-env\lib\site-packages\cate-2.0.0.dev24-py3.6.egg\cate\core\workflow.py", line 980, in _invoke_impl
    return_value = self._op(monitor=monitor, **input_values)
  File "c:\users\prosper\miniconda3\envs\cate-env\lib\site-packages\cate-2.0.0.dev24-py3.6.egg\cate\core\op.py", line 216, in __call__
    return_value = self._wrapped_op(**input_values)
  File "c:\users\prosper\miniconda3\envs\cate-env\lib\site-packages\cate-2.0.0.dev24-py3.6.egg\cate\ops\io.py", line 83, in open_dataset
    monitor=monitor)
  File "c:\users\prosper\miniconda3\envs\cate-env\lib\site-packages\cate-2.0.0.dev24-py3.6.egg\cate\core\ds.py", line 603, in open_dataset
    monitor=monitor.child(80))
  File "c:\users\prosper\miniconda3\envs\cate-env\lib\site-packages\cate-2.0.0.dev24-py3.6.egg\cate\ds\esa_cci_odp.py", line 1008, in make_local
    raise e
  File "c:\users\prosper\miniconda3\envs\cate-env\lib\site-packages\cate-2.0.0.dev24-py3.6.egg\cate\ds\esa_cci_odp.py", line 1001, in make_local
    self._make_local(local_ds, time_range, region, var_names, monitor=monitor)
  File "c:\users\prosper\miniconda3\envs\cate-env\lib\site-packages\cate-2.0.0.dev24-py3.6.egg\cate\ds\esa_cci_odp.py", line 864, in _make_local
    remote_dataset = subset_spatial_impl(remote_dataset, region)
  File "c:\users\prosper\miniconda3\envs\cate-env\lib\site-packages\cate-2.0.0.dev24-py3.6.egg\cate\core\opimpl.py", line 833, in subset_spatial_impl
    lon_min, lat_min, lon_max, lat_max = _pad_extents(ds, extents)
  File "c:\users\prosper\miniconda3\envs\cate-env\lib\site-packages\cate-2.0.0.dev24-py3.6.egg\cate\core\opimpl.py", line 765, in _pad_extents
    lon_min = -180 if lon_min < -180 else lon_min
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()```

### Specifications 

Windows 10, Cate Desktop 2.0.0-dev.24

Evadzi avatar Jan 15 '19 11:01 Evadzi

This error seems to be due to lat lon variables stored as a two dimensional DataArray (mesh) in some CCI datasets e.g. ESACCI-SEAICE-L4-SICONC-AMSR_25.0kmEASE2-NH-20021101-fv2.1.nc.

Have we ever used data like this in cate?

Working format

float lon(lon=720);
  :long_name = "longitude";
  :units = "degrees_east";
  :valid_min = -180.0f; // float
  :valid_max = 180.0f; // float
  :standard_name = "longitude";
  :_ChunkSizes = 720U; // uint

Not Working (Sea Ice lon varibale)

float lon(yc=432, xc=432);
  :long_name = "longitude coordinate";
  :units = "degrees_east";
  :standard_name = "longitude";

double xc(xc=432);
  :long_name = "x coordinate of projection (eastings)";
  :standard_name = "projection_x_coordinate";
  :units = "km";

double yc(yc=432);
  :long_name = "y coordinate of projection (northings)";
  :standard_name = "projection_y_coordinate";
  :units = "km";

dzelge avatar Jan 16 '19 07:01 dzelge

@JanisGailis, FYI

@dzelge The general appoach is to ingest any data. But use of data with unrecognized geocoding/georeferencing should be limmited in Cate, e.g.

  • Cannot display on globe/map
  • subset_spatial should raise ValidationException
  • ...

forman avatar Jan 17 '19 09:01 forman

Ok, as we cannot solve this issue directly (because we don't yet reproject data) we now display better error messages: https://github.com/CCI-Tools/cate/pull/833

forman avatar Jan 18 '19 11:01 forman

Hi, as producer of the CCI Sea Ice data, I would very much like it to be accessible from CATE (the files conform to the CCI data format). Should I open a feature request, or is this "work in progress"?

TomLav avatar Mar 29 '19 22:03 TomLav

Hei, I just discussed this with Carsten at the CCI CMUG workshop. CATE should seamlessly ingest all gridded CCI that follow the Data Standard, including those that are not lat/lon (like the Sea Ice datasets).

What is the status and prospect for reading the Sea Ice data in CATE? (I fully understand the developments were on-hold in between contracts).

Cheers, Thomas

TomLav avatar Nov 07 '19 15:11 TomLav

Dear Cate Team,

I am wondering what is the way forward for the Cate to support datasets that are not in lat/lon grids (like the Sea Ice data). The Sea Ice CCI data will have its Year 1 meeting in about a month and we are eager to discuss visualization in the Toolbox.

Thomas

TomLav avatar Feb 10 '20 08:02 TomLav

@TomLav , apologies for the delay in responding to your earlier questions. You may be able to plot maps from any projection ("plot_map" operation). But you wont be able to display the data on the globe, because the data is not using lon/lat-projection (plate caree). If you don't mind, you can send a sample of your data to me([email protected]) for test. regards, Prosper

Evadzi avatar Feb 10 '20 08:02 Evadzi

Dear Cate Team,

I am wondering what is the way forward for the Cate to support datasets that are not in lat/lon grids (like the Sea Ice data). The Sea Ice CCI data will have its Year 1 meeting in about a month and we are eager to discuss visualization in the Toolbox.

Thomas

@TomLav please open a new issue for this, because this thread is different. Let's then continue discuss ion there. Thanks!

forman avatar Feb 10 '20 09:02 forman

Hei @forman : new issue here: https://github.com/CCI-Tools/cate/issues/869#issue-562436005

TomLav avatar Feb 10 '20 09:02 TomLav