netcdf-c icon indicating copy to clipboard operation
netcdf-c copied to clipboard
verified publisher icon Unidata

"Reference" file system reads (like Kerchunk + Zarr)

Open ashiklom opened this issue 2 years ago • 3 comments

An emerging approach to performant (especially, cloud-native) reads of NetCDF/HDF5 files (as well as GRIB2 and others) is to use a "reference file system" --- basically, a sidecar file that maps a specific byte range within a file onto a chunk that can be read by the Zarr library.

For a more detailed description, see the Kerchunk documentation.

Here's a snippet from inside a reference file system description in JSON format:

        "Rad/0.0.22": [
            "/css/geostationary/BackStage/GOES-17-ABI-L1B-FULLD/2022/001/23/OR_ABI-L1b-RadF-M6C01_G17_s20220012350320_e20220012359386_c20220012359432.nc",
            39046,
            7986
        ],
        "Rad/0.0.23": [
            "/css/geostationary/BackStage/GOES-17-ABI-L1B-FULLD/2022/001/23/OR_ABI-L1b-RadF-M6C01_G17_s20220012350320_e20220012359386_c20220012359432.nc",
            47032,
            7343
        ],
        "Rad/0.0.24": [
            "/css/geostationary/BackStage/GOES-17-ABI-L1B-FULLD/2022/001/23/OR_ABI-L1b-RadF-M6C01_G17_s20220012350320_e20220012359386_c20220012359432.nc",
            11399,
            5174
        ],

Each triplet here corresponds to (1) the target file, (2) the starting byte, and (3) the number of bytes in that chunk. The JSON key (e.g., Rad/0.0.22) describes the chunk, and is exactly equivalent to the path to a chunk blob one would encounter in a typical Zarr store.

Since NetCDF already has the machinery to read Zarr archives via its libnczarr driver, it would be nice to extend that driver to be able to read "virtual" Zarr datasets described in such a format.

In principle, this should be straightforward --- if we already read byte ranges from file systems or S3, NetCDF would just need:

(1) To grab dataset attributes, dimensions, etc. from the JSON file itself (2) To read the binary byte streams based on the triplet references in the JSON (rather than from wherever that information lives in HDF5). (3) Once you have the raw byte streams, everything downstream in the libnczarr implementation --- shuffle, compression, stitching together arrays from chunks, etc. should all be identical to the current libnczarr implementation.

@ksharonin has done some excellent preliminary work on reading chunks given such a reference triplet that should serve as a useful template for getting this implemented in the core NetCDF library: https://github.com/ksharonin/kerchunkC/tree/master/code/c%2B%2B

CC: @amdasilva

ashiklom avatar Oct 23 '23 18:10 ashiklom

One of the strengths of netcdf as an archival format is in that it has been designed from the ground up as a "self describing" file format (in addition to other attributes). It would take some consideration as to how to implement the sort of feature described here (the performance increase is attractive), without losing this fundamental property of netCDF data storage. But it is worth the discussion!

WardF avatar Oct 23 '23 22:10 WardF

In principle, I think this is a good idea. One problem is that it is impossible to interpret the contents of the chunk unless the corresponding metadata is also provided. How do you propose to provide that?

DennisHeimbigner avatar Oct 24 '23 01:10 DennisHeimbigner

So the Kerchunk reference style uses a nested JSON-type structure, basically emulating the entire Zarr directory tree.

Here's a partial example from the Kerchunk docs:

{
  ".zgroup": "{\n    \"zarr_format\": 2\n}",
  ".zattrs": "{\n    \"Conventions\": \"UGRID-0.9.0\n\"}",
  "x/.zattrs": "{\n    \"_ARRAY_DIMENSIONS\": [\n        \"node\"\n ...",
  "x/.zarray": "{\n    \"chunks\": [\n        9228245\n    ],\n    \"compressor\": null,\n    \"dtype\": \"<f8\",\n  ...",
  "x/0": ["s3://bucket/path/file.nc", 294094376, 73825960]
}

Root-level dataset attributes are stored in the root .zattrs key, corresponding exactly to the file-level attributes of a typical NetCDF file. So in this sense, the target JSON reference file is fully self-documenting; it's just that it points to a separate file for the actual data bytes.

Variable-level attributes are stored in a <varname>/.zattrs key. This is where information on dimensions, compression, etc. is stored.

I've included excerpts from a real file describing a large Geostationary dataset in the "Details" below. Note that the root .zarray contains all the information you would normally get from the file-level NetCDF attributes (e.g., "institution": "DOC/NOAA/NESDIS > US Department of Commerce ..."). Note also that the metadata for one of the variables in here, DQF, is stored in "DQF/.zarray" and DQF/.zattrs --- these includes information about the chunk sizes, compression, data type, fill values, total dimensionality, and mapping between dimensions and coordinates (via the _ARRAY_DIMENSIONS field).

        ".zattrs": "{\"Conventions\":\"CF-1.7\",\"LUT_Filenames\":\"SpaceLookParams(FM2A_CDRL79RevP_PR_09_08_04)-681530693.0.h5 QTableBand01(FM2A_C
DRL79RevM_DO_08_00_00)-601637632.0.h5 CalTargetTimeIntervals(FM2A_CDRL79RevP_DO_09_01_00)-653451374.0.h5 BandSaturationLimits(FM2A_CDRL79RevC_DO_07
_00_00)-524728502.0.h5 SolarSpaceLookParams(FM2A_CDRL79RevN_DO_09_00_00)-600765435.0.h5 DeadRowListParams(FM2A_CDRL79RevC_DO_07_00_00)-524728502.0.
h5 Mirror_Record(FM2A_CDRL79RevC_DO_08_00_00)-604635792.0.h5 KalmanAstroConsts(FM2A_CDRL79RevC_DO_07_00_00)-582860861.0.xml KalmanFilterControls(FM
2A_PR_09_08_02)-677655689.0.xml KalmanMeasMaxSensibles(FMAA_INT_ONLY_DO_09_01_00)-653451388.0.xml KalmanPreprocessorControls(FM2A_CDRL79RevC_PR_09_
06_02)-657796800.0.xml KalmanReferenceData(FM2A_CDRL79RevC_DO_07_00_00)-582860861.0.xml KalmanStarCatalogs(FM2A_CDRL79RevC_DO_07_00_00)-582860861.0
.xml ABI_NavigationRDP_Band01(FM2A_CDRL79RevK_DO_08_00_00)-601637632.0.xml ABI_NavigationParameters_Band01(FM2A_CDRL79RevC_DO_07_00_00)-582860861.0
.xml ABI_ResamplingImplementation_Band01(FM2A_CDRL79RevC_DO_07_02_00)-602129336.0.xml ABI_ResamplingParameters_Band01(FM2A_CDRL79RevC_DO_07_00_00)-
582860861.0.xml StarLookParams(FM2A_CDRL79RevC_DO_07_00_00)-524728502.0.h5 StarDetectionParams(FM2A_CDRL79RevC_DO_08_00_00)-594879577.0.xml Resampl
ingScaledConversion(FMAA_INT_ONLY_DO_08_00_00)-1111.0.xml BlockReleaseRegions(FMAA_INT_ONLY_DO_08_00_00)-2222.0.csv VNIR_RetrievalParameters(FM2A_C
DRL79RevC_DO_07_00_00)-524728502.0.h5 SCT_Record(FM2A_CDRL79RevM_DO_09_00_00)-600765435.0.h5 ICM_ConversionConsts(FM2A_CDRL43-18_DO_09_01_00)-65345
1374.0.h5 ICM_SensorCoefficients(FM2A_TMABI_18_159_TMABI_18_533_DO_09_05_00)-677062450.0.h5\",\"Metadata_Conventions\":\"Unidata Dataset Discovery
v1.0\",\"cdm_data_type\":\"Image\",\"dataset_name\":\"OR_ABI-L1b-RadF-M6C01_G17_s20220010000320_e20220010009386_c20220010009424.nc\",\"date_created
\":\"2022-01-01T00:09:42.4Z\",\"id\":\"a02ff0dc-0021-4340-80dc-b6b7f8b63922\",\"institution\":\"DOC\\/NOAA\\/NESDIS > U.S. Department of Commerce,
National Oceanic and Atmospheric Administration, National Environmental Satellite, Data, and Information Services\",\"instrument_ID\":\"FM2\",\"ins
trument_type\":\"GOES-R Series Advanced Baseline Imager (ABI)\",\"iso_series_metadata_id\":\"a70be540-c38b-11e0-962b-0800200c9a66\",\"keywords\":\"
SPECTRAL\\/ENGINEERING > VISIBLE WAVELENGTHS > VISIBLE RADIANCE\",\"keywords_vocabulary\":\"NASA Global Change Master Directory (GCMD) Earth Scienc
e Keywords, Version 7.0.0.0.0\",\"license\":\"Unclassified data.  Access is restricted to approved users only.\",\"naming_authority\":\"gov.nesdis.
noaa\",\"orbital_slot\":\"GOES-West\",\"platform_ID\":\"G17\",\"processing_level\":\"National Aeronautics and Space Administration (NASA) L1b\",\"p
roduction_data_source\":\"Realtime\",\"production_environment\":\"OE\",\"production_site\":\"WCDAS\",\"project\":\"GOES\",\"scene_id\":\"Full Disk\
",\"spatial_resolution\":\"1km at nadir\",\"standard_name_vocabulary\":\"CF Standard Name Table (v35, 20 July 2016)\",\"summary\":\"Single reflecti
ve band ABI L1b Radiance Products are digital maps of outgoing radiance values at the top of the atmosphere for visible and near-IR bands.\",\"time
_coverage_end\":\"2022-01-01T00:09:38.6Z\",\"time_coverage_start\":\"2022-01-01T00:00:32.0Z\",\"timeline_id\":\"ABI Mode 6\",\"title\":\"ABI L1b Ra
diances\"}",

[...]

        "DQF/.zarray": "{\"chunks\":[1,1,226,226],\"compressor\":{\"id\":\"zlib\",\"level\":1},\"dtype\":\"|i1\",\"fill_value\":-1,\"filters\":null
,\"order\":\"C\",\"shape\":[8,144,10848,10848],\"zarr_format\":2}",
        "DQF/.zattrs": "{\"_ARRAY_DIMENSIONS\":[\"bandn\",\"time\",\"y\",\"x\"],\"_Unsigned\":\"true\",\"cell_methods\":\"t: point area: point\",\"
coordinates\":\"band_id band_wavelength t y x\",\"flag_meanings\":\"good_pixel_qf conditionally_usable_pixel_qf out_of_range_pixel_qf no_value_pixe
l_qf focal_plane_temperature_threshold_exceeded_qf\",\"flag_values\":[0,1,2,3,4],\"grid_mapping\":\"goes_imager_projection\",\"long_name\":\"ABI L1
b Radiances data quality flags\",\"number_of_qf_values\":5,\"percent_conditionally_usable_pixel_qf\":4.0000000467443897e-7,\"percent_focal_plane_te
mperature_threshold_exceeded_qf\":0.0,\"percent_good_pixel_qf\":0.9993693828582764,\"percent_no_value_pixel_qf\":1.2399999832268804e-5,\"percent_ou
t_of_range_pixel_qf\":0.00061779998941347,\"standard_name\":\"status_flag\",\"units\":\"1\",\"valid_range\":[0,4]}",

ashiklom avatar Oct 27 '23 20:10 ashiklom