siphon icon indicating copy to clipboard operation
siphon copied to clipboard

Writing out a siphon'd xarray dataset as a netCDF

Open ThomasMGeo opened this issue 1 year ago • 2 comments

Hello! I am trying to write out a netCDF file from a siphon'd xarray dataset. Writing out to Zarr works fine. This may be in the wrong spot (see image).

That is the input:

catUrl = "https://thredds.ucar.edu/thredds/catalog/grib/NCEP/GFS/Global_onedeg_ana/catalog.xml";
datasetName = "Full Collection Dataset";

catalog = TDSCatalog(catUrl)

ds2 = catalog.datasets[datasetName].remote_access(use_xarray=True)
print('ds is a:', type(ds2))

and writing it out:

# Write the second dataset (ds2) to NetCDF format and print the output path
ds2.to_netcdf(paths['ds2'])
print(f"Second dataset written to {paths['ds2']} in NetCDF format.")
  • Problem description: this should explain why the current behavior is a problem and why the expected output is a better solution.

I just wanted to write out a netCDF file, I didn't modify the file to my knowledge. Happier to share a more in depth workflow. The error I get is below:

RuntimeError: NetCDF: Name contains illegal characters

  • Which platform MacOS, python 3.10
  • Versions. Include the output of:

siphon version 0.9 8tg06w

ThomasMGeo avatar Jun 10 '24 22:06 ThomasMGeo

Notebook that has more of the workflow: https://github.com/Unidata/MLscratchpad/blob/main/BlogNotebooks/netCDF_vs_Zarr_filestoage.ipynb

ThomasMGeo avatar Jun 11 '24 15:06 ThomasMGeo

I reproduced this and did notice that

print(ds2.dims)

returns

Frozen({'sigma_layer': 4, '': 2, 'pressure_difference_layer': 1, 'pressure_difference_layer1': 1, 'time': 1, 'isobaric': 41, 'lat': 181, 'lon': 360, 'hybrid': 1, 'isobaric1': 22, 'potential_vorticity_surface': 2, 'sigma': 1, 'altitude_above_msl': 3, 'height_above_ground': 2, 'height_above_ground1': 6})

I think possibly the "Name contains illegal characters" could be referring to a dimension of size 2 with an empty name. Not sure why that dimension has no name when reading in with xarray but is not present when reading with Opendap though.

tdrwenski avatar Jun 26 '24 15:06 tdrwenski