gdal
gdal copied to clipboard
OSGeo
NetCDF: expose 0-dimensional (SCALAR) datasets as metadata
Feature description
Some NetCDF files store important/useful information in the form of 0-dimensional (scalar) datasets. This can be accessed using standard HDF5 tooling, but not in GDAL. I would propose to expose this as additional metadata for the relevant datasets (picked by path similarity) if possible.
Additional context
I've come across this issue working with EUMETSAT's FCI (MTG) files. The fundamental data arrays are readily available using GDAL, but some of the metadata is not directly available.
As an example, consider how GDAL accesses one of the primary subdatasets:
Warning 1: Unhandled X/Y axis unit radian. SRS will ignore axis unit and be likely wrong.
Driver: netCDF/Network Common Data Format
Files: W_XX-EUMETSAT-Darmstadt,IMG+SAT,MTI1+FCI-1C-RRAD-FDHSI-FD--CHK-BODY---NC4E_C_EUMT_20250228124038_IDPFI_OPE_20250228123755_20250228123835_N__O_0076_0034.nc
Size is 5568, 139
Coordinate System is:
PROJCRS["unnamed",
BASEGEOGCRS["unknown",
DATUM["unnamed",
ELLIPSOID["Spheroid",6378137,298.25722356301,
LENGTHUNIT["metre",1,
ID["EPSG",9001]]]],
PRIMEM["Greenwich",0,
ANGLEUNIT["degree",0.0174532925199433,
ID["EPSG",9122]]]],
CONVERSION["Geostationary Satellite (Sweep Y)",
METHOD["Geostationary Satellite (Sweep Y)"],
PARAMETER["Longitude of natural origin",0,
ANGLEUNIT["degree",0.0174532925199433],
ID["EPSG",8802]],
PARAMETER["Satellite Height",35786400,
LENGTHUNIT["metre",1,
ID["EPSG",9001]]],
PARAMETER["False easting",0,
LENGTHUNIT["metre",1],
ID["EPSG",8806]],
PARAMETER["False northing",0,
LENGTHUNIT["metre",1],
ID["EPSG",8807]]],
CS[Cartesian,2],
AXIS["(E)",east,
ORDER[1],
LENGTHUNIT["metre",1,
ID["EPSG",9001]]],
AXIS["(N)",north,
ORDER[2],
LENGTHUNIT["metre",1,
ID["EPSG",9001]]]]
Data axis to CRS axis mapping: 1,2
Origin = (0.155589832847200,0.108868173270957)
Pixel Size = (-0.000055887152603,-0.000055887152603)
Metadata:
/data/ir_105/NC_GLOBAL#long_name=FCI FDHSI Infrared 10.5 micron channel
/data/ir_105/NC_GLOBAL#subsettable=yes
/data/ir_105/quality_channel/NC_GLOBAL#long_name=Quality indicators applicable to a particular channel for the data chunk
/data/ir_105/swath/NC_GLOBAL#long_name=Swath related information
/data/ir_123/NC_GLOBAL#long_name=FCI FDHSI Infrared 12.3 micron channel
/data/ir_123/NC_GLOBAL#subsettable=yes
/data/ir_123/quality_channel/NC_GLOBAL#long_name=Quality indicators applicable to a particular channel for the data chunk
/data/ir_123/swath/NC_GLOBAL#long_name=Swath related information
/data/ir_133/NC_GLOBAL#long_name=FCI FDHSI Infrared 13.3 micron channel
/data/ir_133/NC_GLOBAL#subsettable=yes
/data/ir_133/quality_channel/NC_GLOBAL#long_name=Quality indicators applicable to a particular channel for the data chunk
/data/ir_133/swath/NC_GLOBAL#long_name=Swath related information
/data/ir_38/NC_GLOBAL#long_name=FCI FDHSI Infrared 3.8 micron channel
/data/ir_38/NC_GLOBAL#subsettable=yes
/data/ir_38/quality_channel/NC_GLOBAL#long_name=Quality indicators applicable to a particular channel for the data chunk
/data/ir_38/swath/NC_GLOBAL#long_name=Swath related information
/data/ir_87/NC_GLOBAL#long_name=FCI FDHSI Infrared 8.7 micron channel
/data/ir_87/NC_GLOBAL#subsettable=yes
/data/ir_87/quality_channel/NC_GLOBAL#long_name=Quality indicators applicable to a particular channel for the data chunk
/data/ir_87/swath/NC_GLOBAL#long_name=Swath related information
/data/ir_97/measured/effective_radiance#add_offset=-6.2938299
/data/ir_97/measured/effective_radiance#ancillary_variables=pixel_quality
/data/ir_97/measured/effective_radiance#coordinates=y x
/data/ir_97/measured/effective_radiance#grid_mapping=mtg_geos_projection
/data/ir_97/measured/effective_radiance#long_name=Effective radiance
/data/ir_97/measured/effective_radiance#scale_factor=0.030852109
/data/ir_97/measured/effective_radiance#units=mW.m-2.sr-1.(cm-1)-1
/data/ir_97/measured/effective_radiance#valid_cold_range={0,4095}
/data/ir_97/measured/effective_radiance#valid_range={0,4095}
/data/ir_97/measured/effective_radiance#warm_add_offset=0
/data/ir_97/measured/effective_radiance#warm_scale_factor=0
/data/ir_97/measured/effective_radiance#_FillValue=65535
/data/ir_97/measured/x#add_offset=0.1556177764235012
/data/ir_97/measured/x#axis=X
/data/ir_97/measured/x#long_name=X coordinate in mtg_geos_projection, the packed value corresponds to the column number for each pixel that can be converted into the azimuth angle by using the scale_factor and add_offset of the variable
/data/ir_97/measured/x#scale_factor=-5.58871526031607e-05
/data/ir_97/measured/x#standard_name=projection_x_angular_coordinate
/data/ir_97/measured/x#units=radian
/data/ir_97/measured/x#valid_range={1,5568}
/data/ir_97/measured/y#add_offset=-0.1556177764235012
/data/ir_97/measured/y#axis=Y
/data/ir_97/measured/y#long_name=Y coordinate in mtg_geos_projection, the packed value corresponds to the row number for each pixel that can be converted into the elevation angle by using the scale_factor and add_offset of the variable
/data/ir_97/measured/y#scale_factor=5.58871526031607e-05
/data/ir_97/measured/y#standard_name=projection_y_angular_coordinate
/data/ir_97/measured/y#units=radian
/data/ir_97/measured/y#valid_range={1,5568}
/data/ir_97/NC_GLOBAL#long_name=FCI FDHSI Infrared 9.7 micron channel
/data/ir_97/NC_GLOBAL#subsettable=yes
/data/ir_97/quality_channel/NC_GLOBAL#long_name=Quality indicators applicable to a particular channel for the data chunk
/data/ir_97/swath/NC_GLOBAL#long_name=Swath related information
/data/mtg_geos_projection#coordinates=y x
/data/mtg_geos_projection#grid_mapping_name=geostationary
/data/mtg_geos_projection#inverse_flattening=298.2572235630102
/data/mtg_geos_projection#latitude_of_projection_origin=0
/data/mtg_geos_projection#longitude_of_projection_origin=0
/data/mtg_geos_projection#long_name=MTG geostationary projection
/data/mtg_geos_projection#perspective_point_height=35786400
/data/mtg_geos_projection#semi_major_axis=6378137
/data/mtg_geos_projection#semi_minor_axis=6356752.31424518
/data/mtg_geos_projection#sweep_angle_axis=y
/data/mtg_geos_projection#units=m
/data/nir_13/NC_GLOBAL#long_name=FCI FDHSI Near Infrared 1.3 micron channel
/data/nir_13/NC_GLOBAL#subsettable=yes
/data/nir_13/quality_channel/NC_GLOBAL#long_name=Quality indicators applicable to a particular channel for the data chunk
/data/nir_13/swath/NC_GLOBAL#long_name=Swath related information
/data/nir_16/NC_GLOBAL#long_name=FCI FDHSI Near Infrared 1.6 micron channel
/data/nir_16/NC_GLOBAL#subsettable=yes
/data/nir_16/quality_channel/NC_GLOBAL#long_name=Quality indicators applicable to a particular channel for the data chunk
/data/nir_16/swath/NC_GLOBAL#long_name=Swath related information
/data/nir_22/NC_GLOBAL#long_name=FCI FDHSI Near Infrared 2.2 micron channel
/data/nir_22/NC_GLOBAL#subsettable=yes
/data/nir_22/quality_channel/NC_GLOBAL#long_name=Quality indicators applicable to a particular channel for the data chunk
/data/nir_22/swath/NC_GLOBAL#long_name=Swath related information
/data/vis_04/NC_GLOBAL#long_name=FCI FDHSI Visible 0.4 micron channel
/data/vis_04/NC_GLOBAL#subsettable=yes
/data/vis_04/quality_channel/NC_GLOBAL#long_name=Quality indicators applicable to a particular channel for the data chunk
/data/vis_04/swath/NC_GLOBAL#long_name=Swath related information
/data/vis_05/NC_GLOBAL#long_name=FCI FDHSI Visible 0.5 micron channel
/data/vis_05/NC_GLOBAL#subsettable=yes
/data/vis_05/quality_channel/NC_GLOBAL#long_name=Quality indicators applicable to a particular channel for the data chunk
/data/vis_05/swath/NC_GLOBAL#long_name=Swath related information
/data/vis_06/NC_GLOBAL#long_name=FCI FDHSI Visible 0.6 micron channel
/data/vis_06/NC_GLOBAL#subsettable=yes
/data/vis_06/quality_channel/NC_GLOBAL#long_name=Quality indicators applicable to a particular channel for the data chunk
/data/vis_06/swath/NC_GLOBAL#long_name=Swath related information
/data/vis_08/NC_GLOBAL#long_name=FCI FDHSI Visible 0.8 micron channel
/data/vis_08/NC_GLOBAL#subsettable=yes
/data/vis_08/quality_channel/NC_GLOBAL#long_name=Quality indicators applicable to a particular channel for the data chunk
/data/vis_08/swath/NC_GLOBAL#long_name=Swath related information
/data/vis_09/NC_GLOBAL#long_name=FCI FDHSI Visible 0.9 micron channel
/data/vis_09/NC_GLOBAL#subsettable=yes
/data/vis_09/quality_channel/NC_GLOBAL#long_name=Quality indicators applicable to a particular channel for the data chunk
/data/vis_09/swath/NC_GLOBAL#long_name=Swath related information
/data/wv_63/NC_GLOBAL#long_name=FCI FDHSI Water Vapour 6.3 micron channel
/data/wv_63/NC_GLOBAL#subsettable=yes
/data/wv_63/quality_channel/NC_GLOBAL#long_name=Quality indicators applicable to a particular channel for the data chunk
/data/wv_63/swath/NC_GLOBAL#long_name=Swath related information
/data/wv_73/NC_GLOBAL#long_name=FCI FDHSI Water Vapour 7.3 micron channel
/data/wv_73/NC_GLOBAL#subsettable=yes
/data/wv_73/quality_channel/NC_GLOBAL#long_name=Quality indicators applicable to a particular channel for the data chunk
/data/wv_73/swath/NC_GLOBAL#long_name=Swath related information
NC_GLOBAL#algorithm_version=5.7.5
NC_GLOBAL#cdm_datatype=CDM version 4
NC_GLOBAL#component1=CHK
NC_GLOBAL#component2=BODY
NC_GLOBAL#Conventions=CF-1.7
NC_GLOBAL#count_in_repeat_cycle=0034
NC_GLOBAL#coverage=FD
NC_GLOBAL#[email protected]
NC_GLOBAL#creator_institution=EUMETSAT
NC_GLOBAL#creator_name=EUMETSAT
NC_GLOBAL#creator_type=institution
NC_GLOBAL#creator_url=www.eumetsat.int
NC_GLOBAL#data_designator=IMG+SAT
NC_GLOBAL#data_source=FCI
NC_GLOBAL#date_created=20250228124038
NC_GLOBAL#date_time_position=20250228123000
NC_GLOBAL#disposition_mode=O
NC_GLOBAL#environment=OPE
NC_GLOBAL#facility_or_tool=IDPFI
NC_GLOBAL#format=NC4E
NC_GLOBAL#format_version=FCIL1FS V4C
NC_GLOBAL#geospatial_lat_max=45.23508114269871
NC_GLOBAL#geospatial_lat_min=41.23802758368844
NC_GLOBAL#geospatial_lon_max=75.3206328733758
NC_GLOBAL#geospatial_lon_min=-75.87681850219394
NC_GLOBAL#group_tag=MTI1_FCI_1C_RRAD_FDHSI_2025_059_0076_Original
NC_GLOBAL#history=Original generated file
NC_GLOBAL#institution=EUMETSAT
NC_GLOBAL#instrument_configuration_id=16384
NC_GLOBAL#instrument_configuration_id_version=3
NC_GLOBAL#keywords=MTG FCI Rectified Radiance
NC_GLOBAL#license=https://www.eumetsat.int/legal-framework/data-policy
NC_GLOBAL#linked_data=20250227180155-ser7.tar
NC_GLOBAL#location_indicator=XX-EUMETSAT-Darmstadt
NC_GLOBAL#mtg_name=W_XX-EUMETSAT-Darmstadt,IMG+SAT,MTI1+FCI-1C-RRAD-FDHSI-FD--CHK-BODY---NC4E_C_EUMT_20250228124038_IDPFI_OPE_20250228123755_20250228123835_N__O_0076_0034.nc
NC_GLOBAL#parent_data=MTI1_FCI_1_DPP_GEOSV__2025_059_0075_0000_Original MTI1_FCI_1_DPP_RADIR__2025_058_0000_0000_Original
NC_GLOBAL#platform=MTI1
NC_GLOBAL#processed_count_in_repeat_cycle=0034
NC_GLOBAL#processing_level=1C
NC_GLOBAL#processing_mode=N
NC_GLOBAL#processor_version=5.5.22
NC_GLOBAL#product_id=FCI-1C-RRAD-FDHSI-FD
NC_GLOBAL#project=MTG
NC_GLOBAL#references=www.eumetsat.int
NC_GLOBAL#release_version=Original
NC_GLOBAL#repeat_cycle_in_day=0076
NC_GLOBAL#runtime_data=CONF_MTI1--1-LmkDB_IV_V51B.nc CONF_MTI1--1-LmkCatalog_IV_V51B.xml CONF_MTI1--1-LmkConfig_IV_V51B.xml CONF_MTI1-FCI-1B-StarCatalog_IV_V51B.xml CONF_MTI1-FCI-1B-StarConfig_IV_V51B.xml CONF_MTI1--1-L1cGridConfig_IV_V51B.xml CONF_MTI1-FCI-1C-RadianceEncodingParameters_IV_V51B.xml CONF_MTI1--1-ConstantsParams_IV_V51B.xml CONF_MTI1--1-MiscConfig_IV_V51B.xml CONF_MTI1--1-PhysicalStateVector_IV_V51B.xml CONF_MTI1-FCI-1-Config_IV_V51B.xml CONF_MTI1-INR-1-Config_IV_V51B.xml CONF_MTI1-FCI-1-GeocalConfig_IV_V51B.xml SCCDB-I1-IQT-L1PP_IF_UseCase.xml SCCDB-I1-IDPF-L1PP-IF-8.4.0 FCI CONF_MTI1-FCI-1B-FciAuxData_IV_V51B.nc CONF_MTI1-FCI-1C-RpatConfig_IV_V51B.xml CONF_MTI1--1-GpatConfig_IV_V51B.xml CONF_MTI1-FCI-1C-GpatAux_IV_V51B.nc CONF_MTI1-FCI-1C-DpatConfig_IV_V51B.xml 20250227180155-ser7.dat
NC_GLOBAL#source=IDPFI
NC_GLOBAL#standard_name_vocabulary=CF Standard Name Table v27
NC_GLOBAL#subsettable_groups=vis_04 vis_05 vis_08 vis_09 nir_13 nir_16 wv_63 wv_73 ir_87 ir_97 ir_123 ir_133 vis_06 nir_22 ir_38 ir_105
NC_GLOBAL#subsettable_groups_present=vis_04 vis_05 vis_08 vis_09 nir_13 nir_16 wv_63 wv_73 ir_87 ir_97 ir_123 ir_133 vis_06 nir_22 ir_38 ir_105
NC_GLOBAL#subtype=FDHSI
NC_GLOBAL#summary=Flexible Combined Imager (FCI) Level 1c Rectified Radiance dataset - body data chunk
NC_GLOBAL#time_coverage_duration=PT40S
NC_GLOBAL#time_coverage_end=20250228123835
NC_GLOBAL#time_coverage_resolution=PT40S
NC_GLOBAL#time_coverage_start=20250228123755
NC_GLOBAL#time_position=123000
NC_GLOBAL#title=FCI Level 1C Rectified Radiance Product
NC_GLOBAL#type=RRAD
Geolocation:
SRS=GEOGCRS["unknown",DATUM["unnamed",ELLIPSOID["Spheroid",6378137,298.25722356301,LENGTHUNIT["metre",1,ID["EPSG",9001]]]],PRIMEM["Greenwich",0,ANGLEUNIT["degree",0.0174532925199433,ID["EPSG",9122]]],CS[ellipsoidal,2],AXIS["latitude",north,ORDER[1],ANGLEUNIT["degree",0.0174532925199433,ID["EPSG",9122]]],AXIS["longitude",east,ORDER[2],ANGLEUNIT["degree",0.0174532925199433,ID["EPSG",9122]]]]
X_DATASET=NETCDF:"W_XX-EUMETSAT-Darmstadt,IMG+SAT,MTI1+FCI-1C-RRAD-FDHSI-FD--CHK-BODY---NC4E_C_EUMT_20250228124038_IDPFI_OPE_20250228123755_20250228123835_N__O_0076_0034.nc":/data/ir_97/measured/x
X_BAND=1
Y_DATASET=NETCDF:"W_XX-EUMETSAT-Darmstadt,IMG+SAT,MTI1+FCI-1C-RRAD-FDHSI-FD--CHK-BODY---NC4E_C_EUMT_20250228124038_IDPFI_OPE_20250228123755_20250228123835_N__O_0076_0034.nc":/data/ir_97/measured/y
Y_BAND=1
PIXEL_OFFSET=0
PIXEL_STEP=1
LINE_OFFSET=0
LINE_STEP=1
GEOREFERENCING_CONVENTION=PIXEL_CENTER
Corner Coordinates:
Upper Left ( 0.1555898, 0.1088682) ( 0d 0' 0.01"E, 0d 0' 0.00"N)
Lower Left ( 0.1555898, 0.1010999) ( 0d 0' 0.01"E, 0d 0' 0.00"N)
Upper Right ( -0.1555898, 0.1088682) ( 0d 0' 0.01"W, 0d 0' 0.00"N)
Lower Right ( -0.1555898, 0.1010999) ( 0d 0' 0.01"W, 0d 0' 0.00"N)
Center ( 0.0000000, 0.1049840) ( 0d 0' 0.00"E, 0d 0' 0.00"N)
Band 1 Block=5568x150 Type=UInt16, ColorInterp=Undefined
NoData Value=65535
Unit Type: mW.m-2.sr-1.(cm-1)-1
Offset: -6.29382991790771, Scale:0.0308521091938019
Metadata:
NETCDF_VARNAME=effective_radiance
_FillValue=65535
long_name=Effective radiance
units=mW.m-2.sr-1.(cm-1)-1
valid_range={0,4095}
valid_cold_range={0,4095}
scale_factor=0.030852109
add_offset=-6.2938299
warm_scale_factor=0
warm_add_offset=0
ancillary_variables=pixel_quality
coordinates=y x
grid_mapping=mtg_geos_projection
(the warning is a different matter and most likely deserves its own issue). However:
$ gdalinfo NETCDF:"W_XX-EUMETSAT-Darmstadt,IMG+SAT,MTI1+FCI-1C-RRAD-FDHSI-FD--CHK-BODY---NC4E_C_EUMT_20250228124038_IDPFI_OPE_20250228123755_20250228123835_N__O_0076_0034.nc":/data/ir_97/measured/start_position_rowWarning 1: Variable has 0 dimension(s) - not supported.
ERROR 4: NETCDF:W_XX-EUMETSAT-Darmstadt,IMG+SAT,MTI1+FCI-1C-RRAD-FDHSI-FD--CHK-BODY---NC4E_C_EUMT_20250228124038_IDPFI_OPE_20250228123755_20250228123835_N__O_0076_0034.nc:/data/ir_97/measured/start_position_row: No such file or directory
gdalinfo failed - unable to open 'NETCDF:W_XX-EUMETSAT-Darmstadt,IMG+SAT,MTI1+FCI-1C-RRAD-FDHSI-FD--CHK-BODY---NC4E_C_EUMT_20250228124038_IDPFI_OPE_20250228123755_20250228123835_N__O_0076_0034.nc:/data/ir_97/measured/start_position_row'.
The dataset itself exists, and for example h5ls shows its content without issue:
$ h5ls -d W_XX-EUMETSAT-Darmstadt,IMG+SAT,MTI1+FCI-1C-RRAD-FDHSI-FD--CHK-BODY---NC4E_C_EUMT_20250228124038_IDPFI_OPE_20250228123755_20250228123835_N__O_0076_0034.nc/data/ir_97/measured/start_position_row
start_position_row Dataset {SCALAR}
Data:
4594
I'll point out that GDAL in this case is showing a lot of metadata that belongs to different subdatasets (to wit, all the /data/<other channels which are not ir_97>/*), but it's missing instead the extra information that would be pertinent to the this channel, because it's stored as 0-dimensional datasets. (The pertinence can be extrapolated from the common subpath).
I've come across this issue working with EUMETSAT's FCI (MTG) files
can you provide a link to it ?
The date should be downloadable from here: https://data.eumetsat.int/data/map/EO:EUM:DAT:0665 Or using EUMETSAT's eumdac python script. May require registering with their services.