metview-python icon indicating copy to clipboard operation
metview-python copied to clipboard

Published 20 hours ago • verified publisher icon ecmwf

Unable to complete the notebook "Advanced regridding of data" due to an error during mv.regrid

Open jmb64 opened this issue 9 months ago • 0 comments

What happened?

The rendering of first few plots of the notebook is fine but I get an error message when executing the mv.regrid command (cell 13):

fire_1x1_target_grid = mv.regrid(target_grid, data=fire)

  1. python advanced_regrid.py

2024-05-16 14:37:03,253 INFO Welcome to the CDS 2024-05-16 14:37:03,253 INFO Sending request to https://cds.climate.copernicus.eu/api/v2/resources/cems-fire-historical 2024-05-16 14:37:03,391 INFO Request is completed 2024-05-16 14:37:03,391 INFO Downloading https://download-0005-clone.copernicus-climate.eu/cache-compute-0005/cache/data5/dataset-cems-fire-historical-0bb8bfc5-9cd8-403c-89da-c38fadc3491a.zip to fire.zip (182.5K) 2024-05-16 14:37:04,138 INFO Download rate 244.4K/s
Python - ERROR - 20240516.143735 - Line 0 in 'PythonScript': concept: no match for paramId=255 [ecCodes] Python - ERROR - 20240516.143735 - Line 0 in 'PythonScript': concept: input handle edition=2, centre=ecmf [ecCodes] Python - ERROR - 20240516.143735 - Line 0 in 'PythonScript': Please check the Parameter Database 'https://codes.ecmwf.int/grib/param-db/?id=255' [ecCodes] Python - ERROR - 20240516.143735 - Line 0 in 'PythonScript': Assertion failed: h in HandleDeleter, line 66 of /home/conda/feedstock_root/build_artifacts/metview_1712857313448/work/mir/src/mir/util/Grib.h Traceback (most recent call last): File "/home/jmb001/jmscript/PY/ECMWF/Metview/advanced_regrid.py", line 177, in fire_1x1_target_grid = mv.regrid(target_grid, File "/home/jmb001/.conda/envs/ecmwf/lib/python3.10/site-packages/metview/bindings.py", line 1298, in wrapped return value_from_metview(val) File "/home/jmb001/.conda/envs/ecmwf/lib/python3.10/site-packages/metview/bindings.py", line 1243, in value_from_metview raise retval Exception: Metview error: concept: no match for paramId=255 [ecCodes]

I'm new to Metview so it would be hazardous to offer an explanation. However a few lines in the log (see Relevant Log output below) seems to point towards missing metadata.

NETCDF <UNDEFINED> => (nil) 0 No mapping for NetCDF standard name [<UNDEFINED>] DataInputVariable[name=danger_risk,nc=danger_risk] No mapping for NetCDF standard name [<UNDEFINED>] DataInputVariable[name=danger_risk,nc=danger_risk] Regrid - ERROR - 20240515.181123 - concept: no match for paramId=255 [ecCodes] Regrid - ERROR - 20240515.181123 - concept: input handle edition=2, centre=ecmf [ecCodes] Regrid - ERROR - 20240515.181123 - Please check the Parameter Database 'https://codes.ecmwf.int/grib/param-db/?id=255' [ecCodes] grib_util_set_spec: Failed to change edition to 2: Concept no match Assertion failed: h in HandleDeleter, line 66 of /home/conda/feedstock_root/build_artifacts/metview_1712857313448/work/mir/src/mir/util/Grib.h

The specific input variable (danger_risk) is read from a Netcdf file and might cause a problem for the mapping to the output (GRIB) during the MIR regridding ?

Am I missing some ancillary datafiles related to grib ?

  1. Aside: Another detail caught my attention (since I have not installed the source code for Metview)... this reference to Grib.h include file in a (unknown to me) conda directory

'PythonScript': Assertion failed: h in HandleDeleter, line 66 of /home/conda/feedstock_root/build_artifacts/metview_17
12857313448/work/mir/src/mir/util/Grib.h

What are the steps to reproduce the bug?

I guess run the notebook on the website:

https://metview.readthedocs.io/en/latest/examples/advanced_regrid.html

Version

I've recently install Metview 5.22.0 with conda

Platform (OS and architecture)

cat /etc/os-release --> Ubuntu 22.04.4 LTS uname -a ---> Linux 4.18.0-240.el8.x86_64

Relevant log output

I reran the script after activating debugging information
(export METVIEW_PYTHON_DEBUG=1 ; export MIR_DEBUG=1  ; export MARS_DEBUG=1 )
Here's an excerpt of the listing: 


event - INFO   - 20240515.181123 - Starting service: regrid
event - INFO   - 20240515.181123 - With command    : " /home/jmb001/.conda/envs/ecmwf/lib/metview-bundle/bin/Regrid"
event - INFO   - 20240515.181123 - Got connection
event - INFO   - 20240515.181123 - incoming host is localhost.sack.nb.ec.gc.ca (127.0.0.1)
event - INFO   - 20240515.181123 - Register: regrid jmb001 hpcr5-vis5 1431274 as ref 24
event - INFO   - 20240515.181123 - regrid : Timeout = 300 Alive = 0, Max = 1
event - INFO   - 20240515.181123 - Maximum value for regrid : 1 
event - INFO   - 20240515.181123 - Got connection
event - INFO   - 20240515.181123 - incoming host is localhost.sack.nb.ec.gc.ca (127.0.0.1)
event - INFO   - 20240515.181123 - Register: regrid@1431354 jmb001 hpcr5-vis5 1431354 as ref 25
pool - INFO   - 20240515.181123 - [regrid@1431354] - Object /Process@1430501/PythonScript, line 0:12 [REGRID] not cached
Regrid::serve in
Dataset: pass1...
Dataset: pass1 done
Dataset: pass2...
Assuming that SimpleInputVariable[name=latitude] is a coordinate variable
Assuming that SimpleInputVariable[name=longitude] is a coordinate variable
Assuming that SimpleInputVariable[name=time] is a coordinate variable
Variable 'danger_risk' has both 'missing_value' and '_FillValue' attributes
Variable 'danger_risk: dimension 'time' not listed in coordinates
Variable 'danger_risk: dimension 'latitude' not listed in coordinates
Variable 'danger_risk: dimension 'longitude' not listed in coordinates
Dataset: pass2...
Dataset: pass4...
Dataset: pass4 done
netcdf /fs/site5/eccc/mrd/rpnenv/jmb001/PY/ECMWF/Metview/ECMWF_FWI_DANGER_RISK_20200515_1200_hr_v3.1_con.nc{
dimensions:
        latitude = 721 ;
        longitude = 1440 ;
        time = 1 ;
variables:

        // Kind is data
        // Class is DataInputVariable[name=danger_risk,nc=danger_risk]
        // Coordinates:
        //     CoordinateInputVariable[name=time]
        //     CoordinateInputVariable[name=latitude]
        //     CoordinateInputVariable[name=longitude]

        float danger_risk(time, latitude, longitude) ;
                danger_risk:_FillValue = -9999 ;
                danger_risk:missing_value = -9999 ;
                danger_risk:title = "Danger Risk Class" ;
                danger_risk:units = "-" ;

        // Kind is coordinate
        // Class is CoordinateInputVariable[name=latitude]

        float latitude(latitude) ;
                latitude:axis = "Y" ;
                latitude:long_name = "latitude" ;
                latitude:standard_name = "latitude" ;
                latitude:units = "degrees_north" ;

        // Kind is coordinate
        // Class is CoordinateInputVariable[name=longitude]

        float longitude(longitude) ;
                longitude:axis = "X" ;
                longitude:long_name = "longitude" ;
                longitude:standard_name = "longitude" ;
                longitude:units = "degrees_east" ;

        // Kind is coordinate
        // Class is CoordinateInputVariable[name=time]
        // Codec is GregorianCalendar[reference=2024-05-15 18:11:23, calendar=standard, offset=0]

        float time(time) ;
                time:axis = "T" ;
                time:calendar = "standard" ;
                time:standard_name = "time" ;
                time:units = "days since 2020-05-15 00:00 UTC" ;
// global attributes:
                :CDI = "Climate Data Interface version 1.9.5 (http://mpimet.mpg.de/cdi)" ;
                :CDO = "Climate Data Operators version 1.9.5 (http://mpimet.mpg.de/cdo)" ;
                :Conventions = "CF-1.6" ;
                :ECMWF fire model = "3.1" ;
                :License = "Copernicus" ;
                :NCO = "netCDF Operators version 4.9.2 (Homepage = http://nco.sf.net, Code = http://github.com/nco/nco)" ;
                :Reference date = "20200515" ;
                :cdo_openmp_thread_number = 8 ;
                :history = "Wed Jul 29 18:32:39 2020: cdo -P 8 -z zip_1 -mul ECMWF_FWI_DANGER_RISK_20200515_1200_hr_v3.1_con.tmp era5_hr_lsmask0.01.nc ECMWF_FWI_DANGER_RISK_20200515_1200_hr_v3.1_con.nc
Wed Jul 29 18:32:38 2020: cdo -P 8 -smooth,nsmooth=1,maxpoints=10,radius=0.2deg -remapnn,griddes_hr ECMWF_FWI_DANGER_RISK_20200515_1200_hr_v3.1_con.nc ECMWF_FWI_DANGER_RISK_20200515_1200_hr_v3.1_con.tmp
Wed Jul 29 18:32:37 2020: cdo -f nc4 -selvar,danger_risk ECMWF_FWI_20200515_1200_hr.nc ECMWF_FWI_DANGER_RISK_20200515_1200_hr_v3.1_con.nc
Wed Jul 29 18:32:26 2020: ncatted -O -a version,global,c,c,3.1 ECMWF_FWI_20200515_1200_hr.nc
Wed Jul 29 18:32:26 2020: cdo -f nc4 -z zip_1 -selvar,fwi,ffmc,dmc,dc,isi,bui,dsr,danger_risk ecfire.nc ECMWF_FWI_20200515_1200_hr.nc
Wed Jul 29 18:32:24 2020: ncatted -O -a version,global,c,c,3.1 ecfire.nc" ;
                :version = "3.1" ;

}
NC Field[variable=DataInputVariable[name=danger_risk,nc=danger_risk]]
Variable::lookupCoordinateInputVariable[name=latitude] has standard_name latitude
Variable::lookupCoordinateInputVariable[name=longitude] has standard_name longitude
GRIDSPEC is RegularLL[bbox=90/0/-90/359.75,grid=0.25/0.25,ni=1440,nj=721]
RegularLL[bbox=90/0/-90/359.75,grid=0.25/0.25,ni=1440,nj=721]
{
   "air_pressure_at_sea_level": {
      "paramId": number(151)
   },
   "air_temperature": {
      "paramId": number(167)
   },
   "atmosphere_relative_vorticity": {
      "discipline": number(0),
      "parameterCategory": number(2),
      "parameterNumber": number(12)
   },
   "cloud_area_fraction_assuming_maximum_random_overlap": {
      "paramId": number(164)
   },
   "dew_point_temperature": {
      "paramId": number(168)
   },
   "divergence_of_wind": {
      "discipline": number(0),
      "parameterCategory": number(2),
      "parameterNumber": number(13)
   },
   "eastward_wind": {
      "discipline": number(0),
      "parameterCategory": number(2),
      "parameterNumber": number(2)
   },
   "geopotential": {
      "discipline": number(0),
      "parameterCategory": number(3),
      "parameterNumber": number(4)
   },
   "geopotential_height": {
      "discipline": number(0),
      "parameterCategory": number(3),
      "parameterNumber": number(5)
   },
   "kinetic_energy_dissipation_in_atmosphere_boundary_layer": {
      "discipline": number(0),
      "parameterCategory": number(2),
      "parameterNumber": number(20)
   },
   "lagrangian_tendency_of_air_pressure": {
      "discipline": number(0),
      "parameterCategory": number(2),
      "parameterNumber": number(8)
   },
   "land_binary_mask": {
      "discipline": number(2),
      "parameterCategory": number(0),
      "parameterNumber": number(0),
      "typeOfFirstFixedSurface": number(1)
   },
   "lwe_thickness_of_surface_snow_amount": {
      "paramId": number(141)
   },
   "moisture_content_of_soil_layer": {
      "paramId": number(42)
   },
   "northward_wind": {
      "discipline": number(0),
      "parameterCategory": number(2),
      "parameterNumber": number(3)
   },
   "open_sea_surface_temperature": {
      "paramId": number(34)
   },
   "relative_humidity": {
      "discipline": number(0),
      "parameterCategory": number(1),
      "parameterNumber": number(1)
   },
   "sea_ice_area_fraction": {
      "paramId": number(31)
   },
   "snow_density": {
      "paramId": number(33)
   },
   "soil_temperature": {
      "paramId": number(139)
   },
   "specific_humidity": {
      "discipline": number(0),
      "parameterCategory": number(1),
      "parameterNumber": number(0)
   },
   "surface_air_pressure": {
      "discipline": number(0),
      "parameterCategory": number(3),
      "parameterNumber": number(0),
      "typeOfFirstFixedSurface": number(1)
   },
   "surface_albedo": {
      "paramId": number(243)
   },
   "surface_downward_eastward_stress": {
      "discipline": number(0),
      "parameterCategory": number(2),
      "parameterNumber": number(38)
   },
   "surface_downward_northward_stress": {
      "discipline": number(0),
      "parameterCategory": number(2),
      "parameterNumber": number(37)
   },
   "surface_downwelling_shortwave_flux_in_air": {
      "discipline": number(0),
      "parameterCategory": number(4),
      "parameterNumber": number(7),
      "typeOfFirstFixedSurface": number(1),
      "typeOfStatisticalProcessing": number(1)
   },
   "surface_net_downward_shortwave_flux": {
      "discipline": number(0),
      "parameterCategory": number(4),
      "parameterNumber": number(9),
      "typeOfFirstFixedSurface": number(1),
      "typeOfStatisticalProcessing": number(1)
   },
   "surface_net_upward_longwave_flux": {
      "discipline": number(0),
      "parameterCategory": number(5),
      "parameterNumber": number(5),
      "typeOfFirstFixedSurface": number(1),
      "typeOfStatisticalProcessing": number(1)
   },
   "surface_roughness_length": {
      "discipline": number(2),
      "parameterCategory": number(0),
      "parameterNumber": number(1)
   },
   "surface_temperature": {
      "paramId": number(235)
   },
   "surface_upward_latent_heat_flux": {
      "discipline": number(0),
      "parameterCategory": number(0),
      "parameterNumber": number(10),
      "typeOfFirstFixedSurface": number(1),
      "typeOfStatisticalProcessing": number(1)
   },
   "surface_upward_sensible_heat_flux": {
      "discipline": number(0),
      "parameterCategory": number(0),
      "parameterNumber": number(11),
      "typeOfFirstFixedSurface": number(1),
      "typeOfStatisticalProcessing": number(1)
   },
   "toa_outgoing_longwave_flux": {
      "discipline": number(0),
      "parameterCategory": number(5),
      "parameterNumber": number(5),
      "typeOfFirstFixedSurface": number(8),
      "typeOfStatisticalProcessing": number(1)
   },
   "wind_speed": {
      "paramId": number(49)
   },
   "x_wind": {
      "paramId": number(165)
   },
   "y_wind": {
      "paramId": number(166)
   }
}
NETCDF <UNDEFINED> => (nil) 0
No mapping for NetCDF standard name [<UNDEFINED>] DataInputVariable[name=danger_risk,nc=danger_risk]
Regrid - ERROR  - 20240515.181123 - concept: no match for paramId=255 [ecCodes]
Regrid - ERROR  - 20240515.181123 - concept: input handle edition=2, centre=ecmf [ecCodes]
Regrid - ERROR  - 20240515.181123 - Please check the Parameter Database 'https://codes.ecmwf.int/grib/param-db/?id=255' [ecCodes]
grib_util_set_spec: Failed to change edition to 2: Concept no match
Assertion failed: h in HandleDeleter, line 66 of /home/conda/feedstock_root/build_artifacts/metview_1712857313448/work/mir/src/mir/util/Grib.h
backtrace [1] stack has 17 addresses
(/fs/site5/eccc/mrd/rpnenv/jmb001/micromamba/envs/ecmwf/lib/metview-bundle/bin/../lib/libeckit.so+eckit::BackTrace::dump[abi:cxx11]())0x1a9 
(/fs/site5/eccc/mrd/rpnenv/jmb001/micromamba/envs/ecmwf/lib/metview-bundle/bin/../lib/libeckit.so+eckit::handle_assert(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, eckit::CodeLocation const&))0x534 
(/fs/site5/eccc/mrd/rpnenv/jmb001/micromamba/envs/ecmwf/lib/metview-bundle/bin/../lib/libmir.so+mir::output::GribOutput::save(mir::param::MIRParametrisation const&, mir::context::Context&))0x36df 
(/fs/site5/eccc/mrd/rpnenv/jmb001/micromamba/envs/ecmwf/lib/metview-bundle/bin/../lib/libmir.so+mir::action::io::Save::execute(mir::context::Context&) const)0x16f 
(/fs/site5/eccc/mrd/rpnenv/jmb001/micromamba/envs/ecmwf/lib/metview-bundle/bin/../lib/libmir.so+mir::action::Action::perform(mir::context::Context&) const)0xb4 
(/fs/site5/eccc/mrd/rpnenv/jmb001/micromamba/envs/ecmwf/lib/metview-bundle/bin/../lib/libmir.so+mir::action::ActionPlan::execute(mir::context::Context&) const)0x104 
(/fs/site5/eccc/mrd/rpnenv/jmb001/micromamba/envs/ecmwf/lib/metview-bundle/bin/../lib/libmir.so+mir::action::Job::execute(mir::util::MIRStatistics&) const)0x4e 
(/fs/site5/eccc/mrd/rpnenv/jmb001/micromamba/envs/ecmwf/lib/metview-bundle/bin/../lib/libmir.so+mir::api::MIRJob::execute(mir::input::MIRInput&, mir::output::MIROutput&, mir::util::MIRStatistics&) const)0xf0 
(/home/jmb001/.conda/envs/ecmwf/lib/metview-bundle/bin/Regrid+)0xd32b 
(/home/jmb001/.conda/envs/ecmwf/lib/metview-bundle/bin/Regrid+)0xdef0 
(/fs/site5/eccc/mrd/rpnenv/jmb001/micromamba/envs/ecmwf/lib/metview-bundle/bin/../lib/libMetview.so+MvService::_call_serve(svcid*, request*))0x1d0 
(/fs/site5/eccc/mrd/rpnenv/jmb001/micromamba/envs/ecmwf/lib/metview-bundle/bin/../lib/libmarsclient.so+process_service)0x3e9 
(/fs/site5/eccc/mrd/rpnenv/jmb001/micromamba/envs/ecmwf/lib/metview-bundle/bin/../lib/libmarsclient.so+service_run)0x1a 
(/home/jmb001/.conda/envs/ecmwf/lib/metview-bundle/bin/Regrid+)0x864f 
(/lib/x86_64-linux-gnu/libc.so.6+)0x29d90 
(/lib/x86_64-linux-gnu/libc.so.6+__libc_start_main)0x80 
(/home/jmb001/.conda/envs/ecmwf/lib/metview-bundle/bin/Regrid+)0x86e5 

end of backtrace dump ...
Exception: Assertion failed: h in HandleDeleter, line 66 of /home/conda/feedstock_root/build_artifacts/metview_1712857313448/work/mir/src/mir/util/Grib.h
event - INFO   - 20240515.181123 - Closing service regrid@1431354
event - INFO   - 20240515.181123 - 0 outstanding replies
Python - ERROR  - 20240515.181123 - Line 0 in 'PythonScript': concept: no match for paramId=255 [ecCodes]
Python - ERROR  - 20240515.181123 - Line 0 in 'PythonScript': concept: input handle edition=2, centre=ecmf [ecCodes]
Python - ERROR  - 20240515.181123 - Line 0 in 'PythonScript': Please check the Parameter Database 'https://codes.ecmwf.int/grib/param-db/?id=255' [ecCodes]
Python - ERROR  - 20240515.181123 - Line 0 in 'PythonScript': Assertion failed: h in HandleDeleter, line 66 of /home/conda/feedstock_root/build_artifacts/metview_1712857313448/work/mir/src/mir/util/Grib.h
Traceback (most recent call last):
  File "/home/jmb001/jmscript/PY/ECMWF/Metview/advanced_regrid.py", line 177, in <module>
    fire_1x1_target_grid = mv.regrid(target_grid,
  File "/home/jmb001/.conda/envs/ecmwf/lib/python3.10/site-packages/metview/bindings.py", line 1298, in wrapped
    return value_from_metview(val)
  File "/home/jmb001/.conda/envs/ecmwf/lib/python3.10/site-packages/metview/bindings.py", line 1243, in value_from_metview
    raise retval
Exception: Metview error: concept: no match for paramId=255 [ecCodes]
MetviewInvoker: Closing Metview

Accompanying data

No response

Organisation

Environment and Climate Change Canada

jmb64 avatar May 16 '24 15:05 jmb64