satpy icon indicating copy to clipboard operation
satpy copied to clipboard

VII Reader Fixes and VII Composites Updates

Open cesclc opened this issue 2 months ago • 7 comments

Updates the VII reader stack so recent changes are fully covered and exercised. The reader now leaves the solar-zenith correction to the compositor (i.e., correction was previously performed twice), tolerates multiple filename layouts for L1B NetCDF products, and parses a wider range of sensing start/end timestamps attributes inside NetCDF. Adds the new VII daytime-only and night-time composites along with their YAML wiring, refreshes the reader unit tests to match the calibration and timestamp updates.

  • [x] Contributes to #3202 (not closing it yet due to pending name change)
  • [x] Unit tests for VII readers updated
  • [x] AUTHORS.md updated

cesclc avatar Oct 28 '25 08:10 cesclc

pre-commit.ci autofix

ameraner avatar Oct 28 '25 09:10 ameraner

When I test this with VII_L1B_CrossRef_TDP_V2 it complains about missing key data/calibration_data/band_averaged_solar_irradiance. I can comment that out as a workaround...

However it always crashes with a Segmentation Fault when I try to write an image. Maybe this isn't a fault of the reader, I don't know, but it doesn't crash with any other data. I've tried these tricks and it still crashes:

ulimit -n 4096
export DASK_NUM_WORKERS=1
export OMP_NUM_THREADS=1

howff avatar Nov 06 '25 15:11 howff

As discussed, we should also apply the name change from vii_* to metimage_* (in this PR or another one). We should also update the names here then https://github.com/pytroll/satpy/blob/main/doc/source/examples/vii_l1b_nc.rst.

ameraner avatar Nov 10 '25 12:11 ameraner

When I test this with VII_L1B_CrossRef_TDP_V2 it complains about missing key data/calibration_data/band_averaged_solar_irradiance. I can comment that out as a workaround...

However it always crashes with a Segmentation Fault when I try to write an image. Maybe this isn't a fault of the reader, I don't know, but it doesn't crash with any other data. I've tried these tricks and it still crashes:

ulimit -n 4096
export DASK_NUM_WORKERS=1
export OMP_NUM_THREADS=1

Hi @howff , the KeyError is an incompatibility with the currently public test data. We are now adapting the reader so that it will work with real data. The NetCDF issue is a tricky one - let's keep that discussion in the other issue that you found already.

ameraner avatar Nov 10 '25 12:11 ameraner

Codecov Report

:white_check_mark: All modified and coverable lines are covered by tests. :white_check_mark: Project coverage is 96.34%. Comparing base (293e54e) to head (2554db5). :warning: Report is 36 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #3277      +/-   ##
==========================================
+ Coverage   96.32%   96.34%   +0.01%     
==========================================
  Files         463      463              
  Lines       58896    58916      +20     
==========================================
+ Hits        56729    56760      +31     
+ Misses       2167     2156      -11     
Flag Coverage Δ
behaviourtests 3.60% <0.00%> (+<0.01%) :arrow_up:
unittests 96.43% <100.00%> (+0.01%) :arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

:rocket: New features to boost your workflow:
  • :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

codecov[bot] avatar Dec 02 '25 15:12 codecov[bot]

Codecov is marking that we're not testing the ValueError for unrecognised timestamps - that should be an easy addition, could you maybe quickly add it?

EDIT: I just commited this change myself

ameraner avatar Dec 02 '25 15:12 ameraner

Pull Request Test Coverage Report for Build 19863014968

Details

  • 0 of 0 changed or added relevant lines in 0 files are covered.
  • No unchanged relevant lines lost coverage.
  • Overall first build on main at 96.415%

Totals Coverage Status
Change from base Build 19840971577: 96.4%
Covered Lines: 56611
Relevant Lines: 58716

💛 - Coveralls

coveralls avatar Dec 02 '25 15:12 coveralls