pytroll
Add flash age compositor for li instruments
This Merge request is there to create the flash age compositor related to the li instrument.
- [x] Add the flash_age compositor feature
- [x] Document the code related to the feature
- [x] Test the flash_age compositor
Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:
Project coverage is 96.11%. Comparing base (
3765a5e) to head (01237e2). Report is 293 commits behind head on main.
Additional details and impacted files
@@ Coverage Diff @@
## main #2895 +/- ##
========================================
Coverage 96.10% 96.11%
========================================
Files 381 383 +2
Lines 55544 55673 +129
========================================
+ Hits 53383 53511 +128
- Misses 2161 2162 +1
| Flag | Coverage Δ | |
|---|---|---|
| behaviourtests | 3.89% <0.00%> (-0.01%) |
:arrow_down: |
| unittests | 96.21% <100.00%> (+<0.01%) |
:arrow_up: |
Flags with carried forward coverage won't be shown. Click here to find out more.
:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.
:rocket: New features to boost your workflow:
- :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
![codecov[bot] avatar](https://avatars.githubusercontent.com/in/254?v=4 "Published by a pub.dev verified publisher"){kind=small}
Pull Request Test Coverage Report for Build 10700852657
Details
- 18 of 34 (52.94%) changed or added relevant lines in 1 file are covered.
- No unchanged relevant lines lost coverage.
- Overall first build on add_flash_age_compositor at 96.124%
| Changes Missing Coverage | Covered Lines | Changed/Added Lines | % |
|---|---|---|---|
| satpy/composites/lightning.py | 18 | 34 | 52.94% |
| <!-- | Total: | 18 | 34 |
| Totals | |
|---|---|
| Change from base Build 10654375136: | 96.1% |
| Covered Lines: | 52426 |
| Relevant Lines: | 54540 |
💛 - Coveralls
@ameraner @ClementLaplace any comments about the last comment by @djhoese ?
I would keep this "time-based normalisation" as a compositor, at least for now, since its only use really is inside an image. Since it's also a "made-up" dataset, i.e. it's not present inside the files at all, I'm not sure if it fits the satpy philosophy to put it in a reader (?).
Some issue happened after the merging with the main branch we susperce it is beacuse of PR #2985 the issue that we get is
raceback (most recent call last):
File "/tcenas/home/claplace/project/satpy_script_test/test_flash_age.py", line 17, in <module>
scn.load(['flash_age'])
File "/tcenas/home/claplace/project/satpy/satpy/scene.py", line 1458, in load
self.generate_possible_composites(unload)
File "/tcenas/home/claplace/project/satpy/satpy/scene.py", line 1521, in generate_possible_composites
keepables = self._generate_composites_from_loaded_datasets()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tcenas/home/claplace/project/satpy/satpy/scene.py", line 1540, in _generate_composites_from_loaded_datasets
return self._generate_composites_nodes_from_loaded_datasets(needed_comp_nodes)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tcenas/home/claplace/project/satpy/satpy/scene.py", line 1546, in _generate_composites_nodes_from_loaded_datasets
self._generate_composite(node, keepables)
File "/tcenas/home/claplace/project/satpy/satpy/scene.py", line 1604, in _generate_composite
composite = compositor(prereq_datasets,
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tcenas/home/claplace/project/satpy/satpy/composites/lightning.py", line 106, in __call__
return self._normalize_time(data, new_attrs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tcenas/home/claplace/project/satpy/satpy/composites/lightning.py", line 66, in _normalize_time
data = data.where(data >= begin_time, drop=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tcenas/proj/optcalimg/claplace/virtual_env/satpy-dev/lib/python3.11/site-packages/xarray/core/common.py", line 1225, in where
self = self.isel(**indexers)
^^^^^^^^^^^^^^^^^^^^^
File "/tcenas/proj/optcalimg/claplace/virtual_env/satpy-dev/lib/python3.11/site-packages/xarray/core/dataarray.py", line 1485, in isel
ds = self._to_temp_dataset()._isel_fancy(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tcenas/proj/optcalimg/claplace/virtual_env/satpy-dev/lib/python3.11/site-packages/xarray/core/dataset.py", line 3000, in _isel_fancy
new_var = var.isel(indexers=var_indexers)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tcenas/proj/optcalimg/claplace/virtual_env/satpy-dev/lib/python3.11/site-packages/xarray/core/variable.py", line 1018, in isel
return self[key]
~~~~^^^^^
File "/tcenas/proj/optcalimg/claplace/virtual_env/satpy-dev/lib/python3.11/site-packages/xarray/core/variable.py", line 782, in __getitem__
dims, indexer, new_order = self._broadcast_indexes(key)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tcenas/proj/optcalimg/claplace/virtual_env/satpy-dev/lib/python3.11/site-packages/xarray/core/variable.py", line 622, in _broadcast_indexes
self._validate_indexers(key)
File "/tcenas/proj/optcalimg/claplace/virtual_env/satpy-dev/lib/python3.11/site-packages/xarray/core/variable.py", line 674, in _validate_indexers
raise KeyError(
KeyError: 'Indexing with a boolean dask array is not allowed. This will result in a dask array of unknown shape. Such arrays are unsupported by Xarray.Please compute the indexer first using .compute()'
@mraspaud The test were not working because I did not take in account the dask wrapper into my test, with the new version of my test (see bellow )
def test_flash_age_compositor():
"""Test the flash_age compsitor by comparing two xarrays object."""
comp = LightningTimeCompositor("flash_age",prerequisites=["flash_time"],
standard_name="ligtning_time",
time_range=60,
reference_time="end_time")
attrs_flash_age = {"variable_name": "flash_time","name": "flash_time",
"start_time": datetime.datetime(2024, 8, 1, 10, 50, 0),
"end_time": datetime.datetime(2024, 8, 1, 11, 0, 0),"reader": "li_l2_nc"}
flash_age_value = da.array(["2024-08-01T09:00:00",
"2024-08-01T10:00:00", "2024-08-01T10:30:00","2024-08-01T11:00:00"], dtype="datetime64[ns]")
flash_age = xr.DataArray(
flash_age_value,
dims=["y"],
coords={
"crs": "8B +proj=longlat +ellps=WGS84 +type=crs"
},attrs = attrs_flash_age,name="flash_time")
res = comp([flash_age])
print(res)
expected_attrs = {"variable_name": "flash_time","name": "lightning_time",
"start_time": datetime.datetime(2024, 8, 1, 10, 50, 0),
"end_time": datetime.datetime(2024, 8, 1, 11, 0, 0),"reader": "li_l2_nc",
"standard_name": "ligtning_time"
}
expected_array = xr.DataArray(
da.array([0.0,0.5,1.0]),
dims=["y"],
coords={
"crs": "8B +proj=longlat +ellps=WGS84 +type=crs"
},attrs = expected_attrs,name="flash_time")
xr.testing.assert_equal(res,expected_array)
I'm able to reproduce this issue