Add reader for FY-3E / MERSI-LL

[simonrp84]

This PR adds a reader for the MERSI-LowLight sensor aboard the FengYun-3E satellite. It's currently a draft as there's no additional tests but I aim to add those soon.

• [ ] Tests added
• [ ] Fully documented

[simonrp84]

pre-commit.ci autofix

[codecov[bot]]

Codecov Report

Merging #2124 (7bbf56f) into main (286ec89) will increase coverage by 0.01%. The diff coverage is 97.85%.

@@            Coverage Diff             @@
##             main    #2124      +/-   ##
==========================================
+ Coverage   94.80%   94.81%   +0.01%     
==========================================
  Files         339      339              
  Lines       49198    49391     +193     
==========================================
+ Hits        46641    46832     +191     
- Misses       2557     2559       +2     

Flag	Coverage Δ
behaviourtests	4.41% <0.00%> (-0.02%)	:arrow_down:
unittests	95.44% <97.85%> (+0.01%)	:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
satpy/tests/writer_tests/test_cf.py	99.60% <ø> (ø)
satpy/readers/mersi_l1b.py	97.02% <85.71%> (ø)
satpy/composites/viirs.py	94.47% <90.32%> (-0.23%)	:arrow_down:
satpy/tests/reader_tests/test_mersi_l1b.py	99.77% <99.77%> (ø)
satpy/tests/compositor_tests/test_viirs.py	100.00% <100.00%> (ø)

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

[coveralls]

Coverage: 95.388% (+0.01%) from 95.375% when pulling 7bbf56fabd9909314af4074fcb8f644e1b97a211 on simonrp84:fy3e into 286ec89cd242c0a708ec974f7da82caecc3f0126 on pytroll:main.

[TAlonglong]

Could this be used to have the moon_phase:

>>> import ephem
>>> import datetime
>>> now = datetime.datetime.now()
>>> now
datetime.datetime(2023, 2, 28, 18, 3, 47, 660535)
>>> m = ephem.Moon(now)
>>> m.moon_phase
0.6374500774243032

[TAlonglong]

I get this when reading 250meter resolution data. I will have a look

/home/trygveas/miniconda3/envs/python/lib/python3.9/site-packages/paramiko/transport.py:219: CryptographyDeprecationWarning: Blowfish has been deprecated
  "class": algorithms.Blowfish,
Failed to load DataID(name='6', wavelength=WavelengthRange(min=10.3, central=10.8, max=11.3, unit='µm'), resolution=250, calibration=<calibration.brightness_temperature>, modifiers=()) from <MERSI2L1B: '/home/trygveas/testdata/fy3/FY3E_X_2023_01_11_15_12_A_G_MERSI_0250M_L1B.HDF'>
Traceback (most recent call last):
  File "/home/trygveas/pytroll/satpy/satpy/readers/yaml_reader.py", line 699, in _load_dataset
    projectable = fh.get_dataset(dsid, ds_info)
  File "/home/trygveas/pytroll/satpy/satpy/readers/mersi2_l1b.py", line 124, in get_dataset
    data = self._get_bt_dataset(data, calibration_index, wave_number)
  File "/home/trygveas/pytroll/satpy/satpy/readers/mersi2_l1b.py", line 195, in _get_bt_dataset
    coeffs = self['/attr/TBB_Trans_Coefficient']
  File "/home/trygveas/pytroll/satpy/satpy/readers/hdf5_utils.py", line 96, in __getitem__
    val = self.file_content[key]
KeyError: '/attr/TBB_Trans_Coefficient'

[TAlonglong]

Hm If I do something like this:

        elif self.sensor_name == 'mersi-ll':
            # MERSI-LL stores these coefficients differently
            try:
                coeffs = self['/attr/TBB_Trans_Coefficient']
                corr_coeff_a = coeffs[calibration_index]
                corr_coeff_b = coeffs[calibration_index + N_TOT_IR_CHANS_LL]
            except KeyError:
                return data

I get the data back. But I don't know if there is any science in the data after this.

[djhoese]

Just curious @simonrp84 @TAlonglong what's the status of this PR? My bosses have started asking if I can add this reader soon.

[simonrp84]

@djhoese To be honest this slipped my mind. I added the reader 'for fun' rather than for a specific work project so can't guarantee I'll finish it fast. Am away until 11th April but will look then and give you an estimate of when I might be able to have it done.

[simonrp84]

Looks like recent FY-3E data no longer contains the calibration coefficients (geez, I love consistency!) So, @TAlonglong's "fix" with the try and except does the job it seems.

[simonrp84]

Ok, I updated the names - hopefully the new one is better suited!

[simonrp84]

Should probably rename test_mersi2_l1b.py to test_mersi_l1b.py to match the reader python module, right?

And is this ready to be marked as "Ready for Review"?

Ooops, yes to both. Updated now.

[TAlonglong]

Sorry to interrupt here but I see

h5dump -H /home/trygveas/testdata/fy3/FY3E_X_2023_01_11_15_12_A_G_MERSI_GEO1K_L1B.HDF | grep Moon
      DATASET "MoonAzimuth" {
      DATASET "MoonZenith" {

[simonrp84]

Interesting, how are you retrieving those files? Are they from direct broadcast or the CMA datacentre? All of mine are from CMA - so if yours are DB then maybe there's an inconsistency...

[TAlonglong]

Yeah its from Direct Broadcast processed with the CMA software FY3E package.

[TAlonglong]

I'm fine with that.

[simonrp84]

i've now updated the HNCC composite to cope with missing Moon phase data, it should calculate it internally. Tests are also updated to test as much of this as I can figure out how to do :)

[simonrp84]

Codefactor isn't happy, but those appear to be existing problems rather than something new.

[djhoese]

Yeah that refactor of using fixtures for these reader tests will take a long time. They are all based on VIIRS and MODIS (I think) tests I wrote a long time ago before we were using pytest. They really only work for file handlers using the "utility file handlers" for reading NetCDF and HDF files, but yeah a fixture would likely be much cleaner.