satpy icon indicating copy to clipboard operation
satpy copied to clipboard

Support `group_attrs` in cf_writer

Open zxdawn opened this issue 2 years ago • 5 comments

  • [x] Tests added
  • [x] Fully documented

Sometimes when we save datasets to NetCDF as groups, it will be useful to add the group attributes. In this PR, I add the option by passing group_attrs. It works as same as header_attrs.

Here's an example:

from satpy import Scene
from glob import glob

tropomi_dir = './'
scn = Scene(glob(tropomi_dir+'S5P_OFFL_L2__NO2____20190630T0901*.nc'), reader='tropomi_l2')
vnames = ['nitrogendioxide_tropospheric_column']
scn.load(vnames)

header_attrs = {'description': 'Global attrs'}
group_attrs = {'description': 'Group attrs of data'}

scn.save_datasets(filename='test_cf_group.nc',
                    datasets=vnames,
                    groups={'S5P': vnames},
                    compute=True,
                    header_attrs=header_attrs,
                    group_attrs=group_attrs,
                    writer='cf',
                    engine='netcdf4',
                    )

Output of ncdump -h test_cf_group.nc:

netcdf test_cf_group {

// global attributes:
		:description = "Global attrs" ;
		:history = "Created by pytroll/satpy on 2021-12-02 09:54:54.089499" ;

group: S5P {
  dimensions:
  	time = 1 ;
  	y = 3245 ;
  	x = 450 ;
  	bnds_1d = 2 ;
  variables:
  	int64 time(time) ;
  		time:bounds = "time_bnds" ;
  		time:standard_name = "time" ;
  		time:units = "days since 2019-06-30" ;
  		time:calendar = "proleptic_gregorian" ;
  	float latitude(y, x) ;
  		latitude:_FillValue = NaNf ;
  		latitude:name = "latitude" ;
  		latitude:standard_name = "latitude" ;
  		latitude:units = "degrees_north" ;
  	float longitude(y, x) ;
  		longitude:_FillValue = NaNf ;
  		longitude:name = "longitude" ;
  		longitude:standard_name = "longitude" ;
  		longitude:units = "degrees_east" ;
  	float nitrogendioxide_tropospheric_column(time, y, x) ;
  		nitrogendioxide_tropospheric_column:_FillValue = NaNf ;
  		nitrogendioxide_tropospheric_column:ancillary_variables = "nitrogendioxide_tropospheric_column_precision air_mass_factor_troposphere air_mass_factor_total averaging_kernel" ;
  		nitrogendioxide_tropospheric_column:end_time = "2019-06-30 10:43:01" ;
  		nitrogendioxide_tropospheric_column:file_key = "PRODUCT/nitrogendioxide_tropospheric_column" ;
  		nitrogendioxide_tropospheric_column:file_type = "tropomi_l2" ;
  		nitrogendioxide_tropospheric_column:long_name = "Tropospheric vertical column of nitrogen dioxide" ;
  		nitrogendioxide_tropospheric_column:modifiers = "" ;
  		nitrogendioxide_tropospheric_column:multiplication_factor_to_convert_to_molecules_percm2 = 6.02214e+19f ;
  		nitrogendioxide_tropospheric_column:platform_shortname = "S5P" ;
  		nitrogendioxide_tropospheric_column:sensor = "tropomi" ;
  		nitrogendioxide_tropospheric_column:standard_name = "troposphere_mole_content_of_nitrogen_dioxide" ;
  		nitrogendioxide_tropospheric_column:start_time = "2019-06-30 09:01:31" ;
  		nitrogendioxide_tropospheric_column:units = "mol m-2" ;
  		nitrogendioxide_tropospheric_column:coordinates = "longitude latitude" ;
  	double time_bnds(time, bnds_1d) ;

  // group attributes:
  		:description = "Group attrs of data" ;
  } // group S5P
}

zxdawn avatar Dec 02 '21 10:12 zxdawn

Codecov Report

Merging #1914 (e2e3d6d) into main (f34d563) will increase coverage by 0.00%. The diff coverage is 100.00%.

@@           Coverage Diff           @@
##             main    #1914   +/-   ##
=======================================
  Coverage   93.78%   93.79%           
=======================================
  Files         282      282           
  Lines       42250    42260   +10     
=======================================
+ Hits        39626    39636   +10     
  Misses       2624     2624           
Flag Coverage Δ
behaviourtests 4.74% <0.00%> (-0.01%) :arrow_down:
unittests 94.35% <100.00%> (+<0.01%) :arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
satpy/tests/writer_tests/test_cf.py 99.70% <100.00%> (+<0.01%) :arrow_up:
satpy/writers/cf_writer.py 94.45% <100.00%> (+0.09%) :arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update f34d563...e2e3d6d. Read the comment docs.

codecov[bot] avatar Dec 02 '21 10:12 codecov[bot]

Our new code checker isn't happy with the quality of the 'cf' writer. Any chance you could try cleaning some of this stuff up?

https://github.com/pytroll/satpy/pull/1914/checks?check_run_id=5603372095

djhoese avatar Mar 18 '22 16:03 djhoese

Coverage Status

Coverage increased (+0.001%) to 94.292% when pulling e2e3d6d36e8babef3d75b99131514fe124583006 on zxdawn:cf_group_attrs into f34d56300db61278e588f69f9d6d1b69400ac470 on pytroll:main.

coveralls avatar Mar 18 '22 16:03 coveralls

I have improved the test part. For the cf_writer, the high complexity seems OK?

zxdawn avatar Mar 18 '22 16:03 zxdawn

@zxdawn What is the status of this PR? Any chance of looking into the remaining points?

gerritholl avatar Jul 29 '22 08:07 gerritholl