netcdf-java
netcdf-java copied to clipboard
Unidata
[5.5.2]: Not all ensemble members can not be readed as ens coordinate in GRIB 2 file
Versions impacted by the bug
v5.x
What went wrong?
I have a grid2 data file which contains 4 ensemble members: NCEP, JMA, Beijing and ECMWF. Using ToolUI 5.5.2 only two ensemeble members are extracted with ens coordinate length of 2. The example data file can be downloaded from here: http://www.meteothink.org/downloads/data/201809_12.grib.
The data file information extracted from NCL:
Variable: f Type: file filename: 201809_12 path: 201809_12.grib file global attributes: dimensions: initial_time0_hours = 30 forecast_time0 = 5 lat_0 = 121 lon_0 = 161 variables: float 10u_P1_L103_GLL0 ( initial_time0_hours, forecast_time0, lat_0, lon_0 ) center : US National Weather Service - NCEP (WMC) production_status : TIGGE Operational products long_name : 10 meter u velocity units : m/s _FillValue : 1e+20 grid_type : Latitude/longitude parameter_discipline_and_category : Meteorological products, Momentum parameter_template_discipline_category_number : ( 1, 0, 2, 2 ) level_type : Specified height level above ground (m) level : 10
float 10v_P1_L103_GLL0 ( initial_time0_hours, forecast_time0, lat_0, lon_0 )
center : US National Weather Service - NCEP (WMC)
production_status : TIGGE Operational products
long_name : 10 meter v velocity
units : m/s
_FillValue : 1e+20
grid_type : Latitude/longitude
parameter_discipline_and_category : Meteorological products, Momentum
parameter_template_discipline_category_number : ( 1, 0, 2, 3 )
level_type : Specified height level above ground (m)
level : 10
float 10u_P1_L103_GLL0_1 ( initial_time0_hours, forecast_time0, lat_0, lon_0 )
center : Japanese Meteorological Agency - Tokyo (RSMC)
production_status : TIGGE Operational products
long_name : 10 meter u velocity
units : m/s
_FillValue : 1e+20
grid_type : Latitude/longitude
parameter_discipline_and_category : Meteorological products, Momentum
parameter_template_discipline_category_number : ( 1, 0, 2, 2 )
level_type : Specified height level above ground (m)
level : 10
float 10v_P1_L103_GLL0_1 ( initial_time0_hours, forecast_time0, lat_0, lon_0 )
center : Japanese Meteorological Agency - Tokyo (RSMC)
production_status : TIGGE Operational products
long_name : 10 meter v velocity
units : m/s
_FillValue : 1e+20
grid_type : Latitude/longitude
parameter_discipline_and_category : Meteorological products, Momentum
parameter_template_discipline_category_number : ( 1, 0, 2, 3 )
level_type : Specified height level above ground (m)
level : 10
float 10u_P1_L103_GLL0_2 ( initial_time0_hours, forecast_time0, lat_0, lon_0 )
center : Beijing (RSMC)
production_status : TIGGE Operational products
long_name : 10 meter u velocity
units : m/s
_FillValue : 1e+20
grid_type : Latitude/longitude
parameter_discipline_and_category : Meteorological products, Momentum
parameter_template_discipline_category_number : ( 1, 0, 2, 2 )
level_type : Specified height level above ground (m)
level : 10
float 10v_P1_L103_GLL0_2 ( initial_time0_hours, forecast_time0, lat_0, lon_0 )
center : Beijing (RSMC)
production_status : TIGGE Operational products
long_name : 10 meter v velocity
units : m/s
_FillValue : 1e+20
grid_type : Latitude/longitude
parameter_discipline_and_category : Meteorological products, Momentum
parameter_template_discipline_category_number : ( 1, 0, 2, 3 )
level_type : Specified height level above ground (m)
level : 10
float 10u_P1_L103_GLL0_3 ( initial_time0_hours, forecast_time0, lat_0, lon_0 )
center : European Center for Medium-Range Weather Forecasts
production_status : TIGGE Operational products
long_name : 10 meter u velocity
units : m/s
_FillValue : 1e+20
grid_type : Latitude/longitude
parameter_discipline_and_category : Meteorological products, Momentum
parameter_template_discipline_category_number : ( 1, 0, 2, 2 )
level_type : Specified height level above ground (m)
level : 10
float 10v_P1_L103_GLL0_3 ( initial_time0_hours, forecast_time0, lat_0, lon_0 )
center : European Center for Medium-Range Weather Forecasts
production_status : TIGGE Operational products
long_name : 10 meter v velocity
units : m/s
_FillValue : 1e+20
grid_type : Latitude/longitude
parameter_discipline_and_category : Meteorological products, Momentum
parameter_template_discipline_category_number : ( 1, 0, 2, 3 )
level_type : Specified height level above ground (m)
level : 10
double initial_time0_hours ( initial_time0_hours )
long_name : initial time
units : hours since 1800-01-01 00:00
double initial_time0_encoded ( initial_time0_hours )
long_name : initial time encoded as double
units : yyyymmddhh.hh_frac
float lat_0 ( lat_0 )
long_name : latitude
grid_type : Latitude/Longitude
units : degrees_north
Dj : 0.5
Di : 0.5
Lo2 : 140
La2 : 0
Lo1 : 60
La1 : 60
float lon_0 ( lon_0 )
long_name : longitude
grid_type : Latitude/Longitude
units : degrees_east
Dj : 0.5
Di : 0.5
Lo2 : 140
La2 : 0
Lo1 : 60
La1 : 60
integer forecast_time0 ( forecast_time0 )
long_name : Forecast offset from initial time
units : hours
string initial_time0 ( initial_time0_hours )
long_name : Initial time of first record
units : mm/dd/yyyy (hh:mm)
The data file information extracted from NCDump of ToolUI:
netcdf D:/Temp/grib/201809_12.grib { dimensions: lon = 161; lat = 121; reftime = 30; timeOffset = 5; height_above_ground = 1; ens = 2; variables: int LatLon_Projection; :grid_mapping_name = "latitude_longitude"; :earth_radius = 6371229.0; // double
float lat(lat=121);
:units = "degrees_north";
float lon(lon=161);
:units = "degrees_east";
double reftime(reftime=30);
:units = "Hour since 2018-09-01T12:00:00Z";
:standard_name = "forecast_reference_time";
:long_name = "GRIB reference time";
:calendar = "proleptic_gregorian";
double timeOffset(timeOffset=5);
:_CoordinateAxisType = "TimeOffset";
:units = "Hour";
:standard_name = "forecast_period";
:long_name = "time offset from runtime";
:udunits = "Hour since 2018-09-01T12:00:00Z";
double time(reftime=30, timeOffset=5);
:units = "Hour since 2018-09-01T12:00:00Z";
:standard_name = "time";
:long_name = "GRIB forecast or observation time";
:calendar = "proleptic_gregorian";
:_CoordinateAxisType = "Time";
float height_above_ground(height_above_ground=1);
:units = "m";
:long_name = "Specified height level above ground";
:positive = "up";
:Grib_level_type = 103; // int
:datum = "ground";
int ens(ens=2);
:_CoordinateAxisType = "Ensemble";
float u-component_of_wind_height_above_ground_ens(reftime=30, timeOffset=5, ens=2, height_above_ground=1, lat=121, lon=161);
:long_name = "u-component of wind @ Specified height level above ground";
:units = "m/s";
:description = "u-component of wind";
:missing_value = NaNf; // float
:grid_mapping = "LatLon_Projection";
:coordinates = "reftime time timeOffset ens height_above_ground lat lon ";
:Grib_Variable_Id = "VAR_0-2-2_L103";
:Grib2_Parameter = 0, 2, 2; // int
:Grib2_Parameter_Discipline = "Meteorological products";
:Grib2_Parameter_Category = "Momentum";
:Grib2_Parameter_Name = "u-component of wind";
:Grib2_Level_Type = 103; // int
:Grib2_Level_Desc = "Specified height level above ground";
:Grib2_Generating_Process_Type = "Ensemble forecast";
:Grib2_Statistical_Process_Type = "UnknownStatType--1";
float v-component_of_wind_height_above_ground_ens(reftime=30, timeOffset=5, ens=2, height_above_ground=1, lat=121, lon=161);
:long_name = "v-component of wind @ Specified height level above ground";
:units = "m/s";
:description = "v-component of wind";
:missing_value = NaNf; // float
:grid_mapping = "LatLon_Projection";
:coordinates = "reftime time timeOffset ens height_above_ground lat lon ";
:Grib_Variable_Id = "VAR_0-2-3_L103";
:Grib2_Parameter = 0, 2, 3; // int
:Grib2_Parameter_Discipline = "Meteorological products";
:Grib2_Parameter_Category = "Momentum";
:Grib2_Parameter_Name = "v-component of wind";
:Grib2_Level_Type = 103; // int
:Grib2_Level_Desc = "Specified height level above ground";
:Grib2_Generating_Process_Type = "Ensemble forecast";
:Grib2_Statistical_Process_Type = "UnknownStatType--1";
// global attributes: :Originating_or_generating_Center = "Beijing (RSMC)"; :Originating_or_generating_Subcenter = "0"; :GRIB_table_version = "4,0"; :Type_of_generating_process = "Ensemble forecast"; :file_format = "GRIB-2"; :Conventions = "CF-1.6"; :history = "Read using CDM IOSP GribCollection v3"; :featureType = "GRID"; }
Relevant stack trace
No response
Relevant log messages
No response
If you have an example file that you can share, please attach it to this issue.
If so, may we include it in our test datasets to help ensure the bug does not return once fixed? Note: the test datasets are publicly accessible without restriction.
Yes
Code of Conduct
- [X] I agree to follow the UCAR/Unidata Code of Conduct
Hi Yaqiang:
If you look at that file in ToolsUI Isop/Grib2/Grib2Collection (see attached screenshot), you see that there are indeed 4 records for each forecast time. However, they all have pertN (perturbation number) = 0. Netcdf-Java creates a different ensemble coordinate for each unique combination of PerturbationType and PerturbationNumber. In your file, there are only two: 0,0 and 0,1.
see CoordinateEns.java:
return new EnsCoordValue(pdse.getPerturbationType(), pdse.getPerturbationNumber());
A solution if you can recode the file is to put a different PerturbationNumber (0, 1, 2, 3) for each of the 4 models, for all of the records.
John
Hi John,
Thanks for your information! I will report it to the data provider. But I can not recode the file, so I want to know is it possible to read all 4 ensemble data using NetCDF java library for present data file?
Yaqiang
Hi Yaqiang:
No its not possible with the current version. One could modify the code to include the getNumberEnsembleForecasts() field in the definition of the EnsCoordValue, eg:
new EnsCoordValue(pdse.getPerturbationType(), pdse.getPerturbationNumber(), pdse.getNumberEnsembleForecasts());
However, its not clear if that would break other ensemble models. Unfortunately, the GRIB specification is all but silent on the semantics of the fields it defines. Its left to individual providers and libraries to guess what each other intended.
You could fork the code and build a version of it that would work as you ask. Or petition Unidata and convince them to do that. Or use a special tool to modify the getPerturbationNumber field as I indicated.
Sorry I dont have a better answer.
John
Hi John,
Thanks for your suggestion! ensNumber was added in EnsCoordValue to extract all ensemble members in my case. EnsCoordValue.java, CoordinateEns.java and GribCollectionBuilderFromIndex.java files were revised (https://github.com/Unidata/netcdf-java/compare/maint-5.x...Yaqiang:maint-5.x?expand=1). Now all 4 ensemble coordinate values can be extracted from the data file.
But, you can see my RP that only "public Object extract(Grib2Record gr) " function in CoordinateEns.java can create EnsCoordValue object using the 3th argument with ensemble number. I can't find the value in other EnsCoordValue object creation functions, so I have to replace ensemble number with other values. For example, in readCoord function of GribCollectionBuilderFromIndex.java "i + 1" was used for forged ensemble number.