NCSS grid as point output broken by missing grids

[dopplershift]

A request for both min and max temp from NDFD, such as:

http://thredds.ucar.edu/thredds/ncss/grib/NCEP/NDFD/NWS/CONUS/NOAAPORT/Best?var=Minimum_temperature_height_above_ground_12_Hour_Minimum&var=Maximum_temperature_height_above_ground_12_Hour_Maximum&latitude=40&longitude=-105&time_start=2016-04-23T12%3A00%3A00Z&time_end=2016-05-05T12%3A00%3A00Z&vertCoord=&accept=csv

will return the following CSV:

time,latitude[unit="degrees_north"],longitude[unit="degrees_east"],vertCoord[unit="m"],Minimum_temperature_height_above_ground_12_Hour_Minimum[unit="K"],Maximum_temperature_height_above_ground_12_Hour_Maximum[unit="K"]
2016-04-23T12:00:00Z,40.0,-105.0,2.0,279.8282775878906,NaN
2016-04-24T00:00:00Z,40.0,-105.0,0.0NaN,299.6419982910156
2016-04-24T12:00:00Z,40.0,-105.0,2.0,278.3919982910156,NaN
2016-04-25T00:00:00Z,40.0,-105.0,0.0NaN,292.8512878417969
2016-04-25T12:00:00Z,40.0,-105.0,2.0,279.29022216796875,NaN
2016-04-26T00:00:00Z,40.0,-105.0,0.0NaN,294.802978515625
2016-04-26T12:00:00Z,40.0,-105.0,2.0,279.03948974609375,NaN
2016-04-27T00:00:00Z,40.0,-105.0,0.0NaN,286.5635986328125
2016-04-27T12:00:00Z,40.0,-105.0,2.0,274.9562072753906,NaN
2016-04-28T00:00:00Z,40.0,-105.0,0.0NaN,287.0813903808594
2016-04-28T12:00:00Z,40.0,-105.0,2.0,275.2069091796875,NaN
2016-04-29T00:00:00Z,40.0,-105.0,0.0NaN,284.8240661621094
2016-04-29T12:00:00Z,40.0,-105.0,2.0,277.2021484375,NaN
2016-04-30T00:00:00Z,40.0,-105.0,0.0NaN,281.68939208984375
2016-04-30T12:00:00Z,40.0,-105.0,2.0,274.91290283203125,NaN
2016-05-01T00:00:00Z,40.0,-105.0,0.0NaN,281.97052001953125
2016-05-01T12:00:00Z,40.0,-105.0,2.0,272.5372314453125,NaN
2016-05-02T00:00:00Z,40.0,-105.0,0.0NaN,278.4313049316406
2016-05-02T12:00:00Z,40.0,-105.0,2.0,272.79638671875,NaN
2016-05-03T00:00:00Z,40.0,-105.0,0.0NaN,284.0799865722656
2016-05-03T12:00:00Z,40.0,-105.0,2.0,273.0667419433594,NaN
2016-05-04T00:00:00Z,40.0,-105.0,0.0NaN,289.69927978515625
2016-05-04T12:00:00Z,40.0,-105.0,2.0,275.10614013671875,NaN
2016-05-05T00:00:00Z,40.0,-105.0,0.0NaN,294.5702819824219
2016-05-05T12:00:00Z,40.0,-105.0,2.0,277.5843811035156,NaN

Problems with this:

1. Note the fact that when min temperature (the second to last column) is missing, there is no comma separating the NaN from the previous one
2. Why is the vertical coordinate listed as 0.0 for those same lines? The NCSS HTML page before this said that both min and max temperature are at 2m. I sense this is caused by the NaN as well.

[dopplershift]

More weirdness from same request, but asking for netcdf. Here's the request (I had since removed some times thinking it was related to motherlode issues):

http://thredds.ucar.edu/thredds/ncss/grib/NCEP/NDFD/NWS/CONUS/NOAAPORT/Best?var=Minimum_temperature_height_above_ground_12_Hour_Minimum&var=Maximum_temperature_height_above_ground_12_Hour_Maximum&latitude=40&longitude=-105&time_start=2016-04-28T12%3A00%3A00Z&time_end=2016-05-05T12%3A00%3A00Z&vertCoord=&accept=netcdf

And here's the CDL from ncdump:

netcdf NOAAPORT_Best {
dimensions:
    profile = 15 ;
    station = 1 ;
    height_above_ground = 1 ;
    station_name_strlen = 10 ;
    station_description_strlen = 33 ;
variables:
    float height_above_ground(station, profile, height_above_ground) ;
        height_above_ground:standard_name = "height_above_ground" ;
        height_above_ground:long_name = "height_above_ground" ;
        height_above_ground:units = "m" ;
        height_above_ground:positive = "up" ;
        height_above_ground:axis = "Z" ;
    float Minimum_temperature_height_above_ground_12_Hour_Minimum(station, profile, height_above_ground) ;
        Minimum_temperature_height_above_ground_12_Hour_Minimum:standard_name = "Minimum_temperature_height_above_ground_12_Hour_Minimum" ;
        Minimum_temperature_height_above_ground_12_Hour_Minimum:long_name = "Minimum_temperature_height_above_ground_12_Hour_Minimum" ;
        Minimum_temperature_height_above_ground_12_Hour_Minimum:units = "K" ;
        Minimum_temperature_height_above_ground_12_Hour_Minimum:coordinates = "time longitude latitude height_above_ground" ;
    float Maximum_temperature_height_above_ground_12_Hour_Maximum(station, profile, height_above_ground) ;
        Maximum_temperature_height_above_ground_12_Hour_Maximum:standard_name = "Maximum_temperature_height_above_ground_12_Hour_Maximum" ;
        Maximum_temperature_height_above_ground_12_Hour_Maximum:long_name = "Maximum_temperature_height_above_ground_12_Hour_Maximum" ;
        Maximum_temperature_height_above_ground_12_Hour_Maximum:units = "K" ;
        Maximum_temperature_height_above_ground_12_Hour_Maximum:coordinates = "time longitude latitude height_above_ground" ;
    char station_name(station, station_name_strlen) ;
        station_name:long_name = "station name" ;
        station_name:cf_role = "timeseries_id" ;
    char station_description(station, station_description_strlen) ;
        station_description:long_name = "station description" ;
        station_description:standard_name = "platform_name" ;
    double latitude(station) ;
        latitude:units = "degrees_north" ;
        latitude:long_name = "profile latitude" ;
    double longitude(station) ;
        longitude:units = "degrees_east" ;
        longitude:long_name = "profile longitude" ;
    double time(station, profile) ;
        time:units = "Hour since 2016-04-25T00:00:00Z" ;
        time:calendar = "proleptic_gregorian" ;
        time:standard_name = "time" ;
        time:long_name = "GRIB forecast or observation time" ;

// global attributes:
        :Conventions = "CF-1.6" ;
        :history = "Written by CFPointWriter" ;
        :title = "Extract Points data from Grid file /data/ldm/pub/native/grid/NCEP/NDFD/NWS_noaaport/NDFD_NWS_CONUS_NOAAPORT.ncx3" ;
        :time_coverage_start = "2016-04-28T12:00:00Z" ;
        :time_coverage_end = "2016-05-05T12:00:00Z" ;
        :geospatial_lat_min = 39.9995 ;
        :geospatial_lat_max = 40.0005 ;
        :geospatial_lon_min = -105.0005 ;
        :geospatial_lon_max = -104.9995 ;
        :featureType = "timeSeriesProfile" ;
data:

 height_above_ground =
  2,
  2,
  2,
  2,
  2,
  2,
  2,
  2,
  0,
  0,
  0,
  0,
  0,
  0,
  0 ;

 Minimum_temperature_height_above_ground_12_Hour_Minimum =
  NaNf,
  NaNf,
  NaNf,
  NaNf,
  NaNf,
  NaNf,
  NaNf,
  277.5844,
  0,
  0,
  0,
  0,
  0,
  0,
  0 ;

 Maximum_temperature_height_above_ground_12_Hour_Maximum =
  284.8241,
  281.6894,
  281.9705,
  278.4313,
  284.08,
  289.6993,
  294.5703,
  NaNf,
  0,
  0,
  0,
  0,
  0,
  0,
  0 ;

 station_name =
  "Grid Point" ;

 station_description =
  "Grid Point at lat/lon=40.0,-105.0" ;

 latitude = 40 ;

 longitude = -105 ;

 time =
  84, 108, 132, 156, 180, 204, 228, 252, 0, 0, 0, 0, 0, 0, 0 ;
}

Note the:

1. Values of 0 in time, data variables, and vert coord
2. Difference in time units vs. time_coverage metadata

(Also note full machine path in title metadata. Oops.) If you change the request and get either (but not both) minimum or maximum temperature, everything is fine.

I've attached the file if anyone's curious. NOAAPORT_Best.nc.gz

[cwardgar]

I've confirmed that this problem still exists on THREDDS Main. I'd like to work on it, but I need to be able to reproduce it locally first. Can somebody grab the data and the relevant catalog snippet? I'd do it myself, but I don't have access to our production machines.

[lesserwhirls]

We really need to fix the issue of you not having access on our machines :-/

You should be able to grab a few from here via the http service. Here are the latest three:

NDFD_NWS_CONUS_2p5km_20180322_0000.grib2 NDFD_NWS_CONUS_2p5km_20180322_1200.grib2 NDFD_NWS_CONUS_2p5km_20180323_0000.grib2

and the config catalog for this collection can be found here.

[cwardgar]

Okay, so I attempted to reproduce with the datasets Sean provided and the following catalog:

<?xml version="1.0" encoding="UTF-8"?>
<catalog xmlns="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0"
         xmlns:xlink="http://www.w3.org/1999/xlink"
         name="Unidata THREDDS Data Server - NCEP models"
         version="1.0.7">
    
    <dataset name="National Digital Forecast Database (NDFD)">
        <featureCollection name="National Weather Service CONUS Forecast Grids (NOAAPORT)"
                           featureType="GRIB2" harvest="true" path="grib/NCEP/NDFD/NWS/CONUS/NOAAPORT">
            <collection spec="/Users/cwardgar/Downloads/.*grib2$"
                        dateFormatMark="#5km_#yyyyMMdd_HHmm"
                        timePartition="file"
                        name="NDFD_NWS_CONUS_NOAAPORT"
                        olderThan="5 min"/>
        </featureCollection>
    </dataset>
</catalog>

That catalog was fine and I was able to get to NCSS. Then I submitted this request:

http://localhost:8081/thredds/ncss/grid/grib/NCEP/NDFD/NWS/CONUS/NOAAPORT/Best?var=Maximum_temperature_height_above_ground_12_Hour_Maximum&var=Minimum_temperature_height_above_ground_12_Hour_Minimum&latitude=40&longitude=-105&time_start=2018-03-25T06:00:00Z&time_end=2018-03-31T00:00:00Z&timeStride=1&vertCoord=&accept=csv

The result? Well...

ucar.ma2.InvalidRangeException: time2: subsetValuesDiscontinuous not done yet;
        at ucar.nc2.grib.coverage.GribCoverageDataset.readData(GribCoverageDataset.java:1117)
        ...
ucar.ma2.InvalidRangeException: time3: subsetValuesDiscontinuous not done yet;
        at ucar.nc2.grib.coverage.GribCoverageDataset.readData(GribCoverageDataset.java:1117)
        ...

It looks like I'll be working on #1054 first. At least I have a local repro for it now.