satpy
satpy copied to clipboard
pytroll
Cannot load ABI L1 datasets from Scene combining L1 and L2
Describe the bug
When I load ABI L1 and L2 data into the same Scene, I can't load L1 datasets.
To Reproduce
from satpy import Scene
from satpy.utils import debug_on; debug_on()
files_l1b = ["/media/nas/x21308/abi/G16-ABI-L1B-RadC/2021/OR_ABI-L1b-RadC-M6C10_G16_s20213332001177_e20213332003561_c20213332004003.nc"]
files_l2 = ["/media/nas/x21308/abi/G16-L2-ACHAC/OR_ABI-L2-ACHAC-M6_G16_s20213332001177_e20213332003550_c20213332007025.nc"]
global_scene = Scene({"abi_l1b": files_l1b, "abi_l2_nc": files_l2})
global_scene.load(["C10"])
Expected behavior
I expect the loading to work just as it would (and does) when I pass only L1 files upon Scene initiation.
Actual results
It fails to load channel 10 because it says the file types abi_l2_cmip_c10 and abi_l2_cmip are missing. That's wrong. I shouldn't and don't need L2 CMIP data to load L1 channel 10.
[DEBUG: 2021-12-01 14:08:30 : satpy.readers.yaml_reader] Reading ('/data/gholl/checkouts/satpy/satpy/etc/readers/abi_l1b.yaml',)
[DEBUG: 2021-12-01 14:08:30 : satpy.readers.yaml_reader] Assigning to abi_l1b: ['/media/nas/x21308/abi/G16-ABI-L1B-RadC/2021/OR_ABI-L1b-RadC-M6C10_G16_s20213332001177_e20213332003561_c20213332004003.nc']
[DEBUG: 2021-12-01 14:08:31 : satpy.readers.yaml_reader] Reading ('/data/gholl/checkouts/satpy/satpy/etc/readers/abi_l2_nc.yaml',)
[DEBUG: 2021-12-01 14:08:31 : satpy.readers.yaml_reader] Assigning to abi_l2_nc: ['/media/nas/x21308/abi/G16-L2-ACHAC/OR_ABI-L2-ACHAC-M6_G16_s20213332001177_e20213332003550_c20213332007025.nc']
[DEBUG: 2021-12-01 14:08:31 : satpy.composites.config_loader] Looking for composites config file abi.yaml
[DEBUG: 2021-12-01 14:08:31 : satpy.composites.config_loader] Looking for composites config file visir.yaml
[WARNING: 2021-12-01 14:08:31 : satpy.readers.yaml_reader] Required file type '['abi_l2_cmip_c10', 'abi_l2_mcmip']' not found or loaded for 'C10'
[WARNING: 2021-12-01 14:08:31 : satpy.scene] The following datasets were not created and may require resampling to be generated: DataID(name='C10', wavelength=WavelengthRange(min=7.24, central=7.34, max=7.44, unit='µm'), calibration=<calibration.brightness_temperature>, modifiers=())
Resampling does not generate C10 either.
Environment Info:
- OS: openSUSE Leap 15.3
- Satpy Version: v0.31.0-158-g5dae1e6c
Oh very interesting. So the problem is that the ABI L2 reader has a C10 that it can load:
https://github.com/pytroll/satpy/blob/4950ccde2304d82fb7463289febe17f86a232fc4/satpy/etc/readers/abi_l2_nc.yaml#L80-L85
If you asked for the "calibration='radiance'" version of that channel I would suspect it to work. I'm not sure there is an easy way to fix this. It would require the portion of the Scene/DependencyTree that is asking the reader for a dataset to check all readers that know about a dataset and attempt to load the dataset from each one until it is successful (or fails).
The only issue with that is then they can't be used when making composites as most of the composites use the specific product name.
Wow, can't believe this issue already exists. @simonrp84 brought this up on slack which I'm guessing is why you commented here. In his case he was able to get a L1b + L2 composite to work when he updated the composites using L1b channels to specify a resolution: X since the L2 reader's DataID doesn't include resolution, but the L1b reader does. This is a workaround, but only lucky in this particular case since logically the L2 reader should be able (allowed) to be specify the resolution. Also important to note, he was not provided the L2 files that provided the equivalent of the L1b channels so the dependency tree wanted to use DataIDs that weren't available/loadable.
The main issue(s) I discovered after last diving into the dependency tree code is that the dependency tree looks at all known DataIDs and doesn't consider whether or not the DataIDs are available. Also the dep tree says "here are all the DataIDs from all the readers that match the request, which one matches most closely". In the L1b/L2 case, just by chance of the way it was coded, when a resolution isn't specified in the request, the resolution in the DataID is ignored and the L2 (no resolution) DataID is considered a "better match". If the L2 DataID did define a resolution I have a feeling we'd get a TooManyResults exception meaning that the dependency tree found two or more DataIDs that matched equally well to the requested DataQuery.
Regardless of the above, even if L2 specified resolution for its DataIDs and even if the user provided the necessary L2 data files, the dependency tree should probably be able to resolve/choose these conflicts based on:
- What's available. For example, L2's version isn't available but L1b is, use L1b.
- Some inherent order or user provided order. The user may initialize the Scene with a
filenames={"reader": [...], "reader2": [...]}dictionary where the dict has some order associated with it depending on how it was created. Or the readers could have some relationship/understanding that the L2 reader is reading "descendant" datasets of the L1b data. I'm not sure if in that case the dependency tree should prefer L2 (the descendant/child) or L1b (the ancestor/parent).
On your second point, I think that if both L1 and L2 dataset files are provided by the user them the higher level dataset should be used by default.
