Missing GRIB-1 time range indicators

[rschmunk]

Versions impacted by the bug

I am working with a devo copy of netCDF-Java 5.5.4, snapshot downloaded early August.

What went wrong?

A user pinged me because netCDF-Java (and thus Panoply) would not open a GRIB-1 dataset, with an exception thrown because the data use time range indicator 130. See stack trace below.

In looking at the netCDF-Java code, I see that Grib1ParamTime rejects and time range indicator with value > 125. However, in looking for more info about this, I found that there are at least thirteen time range indicators greater than that limit, as shown on e.g. this NOAA/NCEP webpage

Although the above link is to a NOAA webpage, the user originally downloaded the data from an EU Copernicus site. Unfortunately, I don't know what specific field input he used there to acquire the problem dataset. Instead he provided me a copy of the file.

Is there any chance that netCDF-Java developers can look into this in the foreseeable future? I haven't looked at the code in Grib1ParamTime closely enough to see if I could puzzle it out for myself, and bugs in another app are calling for my immediate attention.

Relevant stack trace

Caused by: java.lang.IllegalArgumentException: PDS: Unknown Time Range Indicator 130
	at ucar.nc2.grib.grib1.Grib1ParamTime.<init>(Grib1ParamTime.java:399)
	at ucar.nc2.grib.grib1.tables.Grib1Customizer.getParamTime(Grib1Customizer.java:118)
	at ucar.nc2.grib.grib1.Grib1Record.getParamTime(Grib1Record.java:124)
	at ucar.nc2.grib.collection.Grib1CollectionBuilder.filterIntervals(Grib1CollectionBuilder.java:160)
	at ucar.nc2.grib.collection.Grib1CollectionBuilder.makeGroups(Grib1CollectionBuilder.java:107)
	at ucar.nc2.grib.collection.GribCollectionBuilder.createMultipleRuntimeCollections(GribCollectionBuilder.java:128)
	at ucar.nc2.grib.collection.GribCollectionBuilder.createIndex(GribCollectionBuilder.java:120)
	at ucar.nc2.grib.collection.GribCdmIndex.openGribCollectionFromDataFile(GribCdmIndex.java:821)
	at ucar.nc2.grib.collection.GribCdmIndex.openGribCollectionFromDataFile(GribCdmIndex.java:804)
	at ucar.nc2.grib.collection.GribCdmIndex.openGribCollectionFromRaf(GribCdmIndex.java:774)
	at ucar.nc2.grib.collection.GribIosp.build(GribIosp.java:183)
	at ucar.nc2.NetcdfFiles.build(NetcdfFiles.java:815)
	at ucar.nc2.NetcdfFiles.open(NetcdfFiles.java:754)
	at ucar.nc2.NetcdfFiles.open(NetcdfFiles.java:276)

Relevant log messages

No response

If you have an example file that you can share, please attach it to this issue.

If so, may we include it in our test datasets to help ensure the bug does not return once fixed? Note: the test datasets are publicly accessible without restriction.

Yes

Code of Conduct

• [X] I agree to follow the UCAR/Unidata Code of Conduct

[JohnLCaron]

Hi Robert:

The way that works is that a Grib1Customizer customizes behavior when its not standard WMO GRib1. So if you look at NcepTables you see it does handle timeRangeIndicator=130. OTOH, if its copernicus, its probably not NCEP so you'll have to track down the correct tables, since theres no reason to assume that they are using the same definition as NCEP. Which is why we cant just "look around" for some table with timeRangeIndicator=130.

In any case, you have to look at the grib record, figure out what table it needs, then configure the code to recognize what table to use for that file:

https://docs.unidata.ucar.edu/netcdf-java/current/userguide/grib_tables.html#grib-1-table-handling

If you want to post the file, Im sure the Unidata folks can help. If its huge, and you know how (ToolsUI can do it, but its obscure), you can extract a sample record that shows the problem.

Regards, John

On Sun, Oct 8, 2023 at 8:15 PM R. Schmunk @.***> wrote:

Versions impacted by the bug

I am working with a devo copy of netCDF-Java 5.3.4, snapshot downloaded early August. What went wrong?

A user pinged me because netCDF-Java (and thus Panoply) would not open a GRIB-1 dataset, with an exception thrown because the data use time range indicator 130. See stack trace below.

In looking at the netCDF-Java code, I see that Grib1ParamTime rejects and time range indicator with value > 125. However, in looking for more info about this, I found that there are at least thirteen time range indicators greater than that limit, as shown on e.g. this NOAA/NCEP webpage https://www.nco.ncep.noaa.gov/pmb/docs/on388/table5.html

Although the above link is to a NOAA webpage, the user originally downloaded the data from an EU Copernicus site https://cds.climate.copernicus.eu/cdsapp#!/dataset/reanalysis-era5-single-levels-monthly-means?tab=overview. Unfortunately, I don't know what specific field input he used there to acquire the problem dataset. Instead he provided me a copy of the file.

Is there any chance that netCDF-Java developers can look into this in the foreseeable future? I haven't looked at the code in Grib1ParamTime closely enough to see if I could puzzle it out for myself, and bugs in another app are calling for my immediate attention. Relevant stack trace

Caused by: java.lang.IllegalArgumentException: PDS: Unknown Time Range Indicator 130 at ucar.nc2.grib.grib1.Grib1ParamTime.(Grib1ParamTime.java:399) at ucar.nc2.grib.grib1.tables.Grib1Customizer.getParamTime(Grib1Customizer.java:118) at ucar.nc2.grib.grib1.Grib1Record.getParamTime(Grib1Record.java:124) at ucar.nc2.grib.collection.Grib1CollectionBuilder.filterIntervals(Grib1CollectionBuilder.java:160) at ucar.nc2.grib.collection.Grib1CollectionBuilder.makeGroups(Grib1CollectionBuilder.java:107) at ucar.nc2.grib.collection.GribCollectionBuilder.createMultipleRuntimeCollections(GribCollectionBuilder.java:128) at ucar.nc2.grib.collection.GribCollectionBuilder.createIndex(GribCollectionBuilder.java:120) at ucar.nc2.grib.collection.GribCdmIndex.openGribCollectionFromDataFile(GribCdmIndex.java:821) at ucar.nc2.grib.collection.GribCdmIndex.openGribCollectionFromDataFile(GribCdmIndex.java:804) at ucar.nc2.grib.collection.GribCdmIndex.openGribCollectionFromRaf(GribCdmIndex.java:774) at ucar.nc2.grib.collection.GribIosp.build(GribIosp.java:183) at ucar.nc2.NetcdfFiles.build(NetcdfFiles.java:815) at ucar.nc2.NetcdfFiles.open(NetcdfFiles.java:754) at ucar.nc2.NetcdfFiles.open(NetcdfFiles.java:276)

Relevant log messages

No response If you have an example file that you can share, please attach it to this issue.

If so, may we include it in our test datasets to help ensure the bug does not return once fixed? Note: the test datasets are publicly accessible without restriction.

Yes Code of Conduct

• I agree to follow the UCAR/Unidata Code of Conduct

— Reply to this email directly, view it on GitHub https://github.com/Unidata/netcdf-java/issues/1246, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAEVZQCZBVNMJ5Z6I3PIYDTX6NM5XAVCNFSM6AAAAAA5YD2PESVHI2DSMVQWIX3LMV43ASLTON2WKOZRHEZTEMJZGQZDSNI . You are receiving this because you are subscribed to this thread.Message ID: @.***>

[rschmunk]

@JohnLCaron, Depends on your definition of huge. The sample file I was provided is 170 MB.

[JohnLCaron]

hmm i dont see the file

[rschmunk]

@JohnLCaron, I didn't include a link in my original post about this. Here's a Dropbox link.

[rschmunk]

And today I get a complaint from someone downloading data from the same Copernicus website that they are getting an "Unknown Time Range Indicator 128" exception trying to open the download. Sorry, no sample file provided this time.

[JohnLCaron]

Its clear that ECMWF has started using some non-standard grib1 time parameters. Someone has to go find out what they are, and perhaps also check on whether the parameter tables have changed. Its fair to ask your customer to do so, presumably they are the ones who care, and perhaps might know enough to tell if they are right. Then send that information into Unidata support, who should be able to add the code to implement it.

This will happen for as long as you try to support GRIB (esp GRIB1), so you could try to standardize it, and point people at a FAQ or something. Or you could stop trying to support GRIB1.

We did make a deliberate decision to let things break, rather than allow incorrect data. But we could add a debug flag allowing people to see the data, and set the time range to something like Jan 1, 1970.

It also occurs to me that you are looking at a legacy product for which there is a newer, maybe GRIB2 equivalent.

On Wed, Oct 11, 2023 at 9:30 AM R. Schmunk @.***> wrote:

And today I get a complaint from someone downloading data from the same Copernicus website that they are getting an "Unknown Time Range Indicator 128" exception trying to open the download. Sorry, no sample file provided this time.

— Reply to this email directly, view it on GitHub https://github.com/Unidata/netcdf-java/issues/1246#issuecomment-1757962502, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAEVZQH7SO4RWOAXXETYPTDX623PPANCNFSM6AAAAAA5YD2PEQ . You are receiving this because you were mentioned.Message ID: @.***>