Add missing EUMeTrain RGBs for SEVIRI and FCI

[gerritholl]

Feature Request

Is your feature request related to a problem? Please describe.

I would like to be able to generate all the "classic" EUMeTrain composites (11 RGBs and the Sandwich product) using both SEVIRI and FCI. Currently, one RGB (24-hour microphysics) is missing entirely, four composites are missing for FCI, and three have non-matching implementations.

Satpy is increasingly popular and when Satpy implements an RGB as a composite in a certain way, it may well become the de facto standard. At the same time, RGB communities from CIRA, NOAA, EUMETSAT, EUMeTrain, JMA, and others, are getting together in workshops to agree on standardised composite definitions. Where such a consensus exists, it would be desirable if this exact definition would work out-of-the-box in Satpy.

I went through the EUMeTrain RGBs to compare the Satpy implementation with the EUMeTrain definitions and describe what is absent or present, and whether definitions match between Satpy and EUMeTrain:

Composite/RGB	SEVIRI	FCI	comment
Airmass (mid-latitude)	yes	yes
Airmass (tropical)	differs	differs
Dust	yes	yes
24-hour microphysics	yes	yes	Similar to Ash RGB with different stretching / gamma correction. Similar to night microphysics, but uses 8.7 µm rather than 3.9 µm. Introduced for FCI in #2780.
Ash	yes	yes
Day microphysics	yes	yes	Satpy additionally has a "winter" version, for which I could find no definition in the RGB community. It uses different stretching or gamma correction.
Severe storms (mid-latitude)	missing/differs	missing/differs	There is a Satpy "HRV Severe Storms RGB", that is completely different. EUMeTrain notes that this RGB is sometimes called "convection", but the Satpy convection RGB has different stretching and gamma correction.
Severe storms (tropical)	missing	missing
Snow	yes	yes	Introduced for FCI in #2780
Natural colour	differs	differs
HRV Clouds	yes	missing	May need work or not work for FCI. The CIRA Day Cloud Convection RGB is identical to the HRV Clouds RGB, except HRV is replaced by 0.6 µm. The CIRA Day Cloud Convection RGB is not implemented in Satpy.
HRV Fog	yes	missing	May need work for FCI. Not to be confused with the entirely different "Fog RGB". Similar to natural colour but with different colours.
Night microphysics (mid-latitude)	yes	yes
Night microphysics (tropical)	missing	missing
IR Sandwich	yes	yes	Introduced for FCI in #2780

Describe the solution you'd like

I would like composite definitions for all EUMeTrain products that work for both SEVIRI and FCI.

Describe any changes to existing user workflow

Missing composites can be added, but diverging definitions would be difficult to change without causing problems for operational production. Where definitions differ, we may have to resort to adding a new composite with the tag emt_ prepended. I don't think we should change the existing composites before we introduce breaking changes with Satpy 1.0, whenever that may be.

Additional context

Satpy is increasingly popular to the degree that when Satpy gives an RGB/composite a certain name, this might become the de-facto standard. Therefore, it is important that when the RGB community does reach a consensus, that Satpy should consider implementing this.

See also:

• EUMeTrain RGB recipes
• October 2022 RGB Workshop Final Report
• JMA HIMAWARI
• CIRA VISIT (ABI)
• NASA Sport VIIRS RGBs

Definitions used by other agencies may differ from the ones used by EUMeTrain.

Of course, I could make all those implementations locally, but I think the standard ones are worth having in Satpy with definitions that match the standard.

I have not considered the new composites that are possible with FCI, but not with SEVIRI, because I don't know if there is a community consensus yet on how they should be defined: True Colour RGB, Geocolor, Cloud Phase RGB, Cloud Type RGB, Fire Temperature RGB, or yet-to-be-named RGBs such as ones that include the solar moisture transmittance (SMT) using the 0.91 µm/0.86 µm ratio. Those should also be implemented in Satpy, but probably with a warning that definitions may change as community consensus on the definition develops.

[pnuu]

I'd forget the HRV composites for FCI, there's no panchromatic channel. If one really wants to play with it, averaging the vis_05, vis_06 and vis_08 channels would give roughly the same overall range, but have a spectral gap from 690 nm to 815 nm, which is 1/4th of the range.

[ameraner]

Thank you for this analysis! Regarding the new FCI composites: at EUM we have indeed a few recipes that we have been experimenting with in the last months for the various public releases. As discussed in Slack, we will try to push them to satpy during the upcoming PCW. As we recently saw for the True Color, there's a clear benefit for the community to start using them so that we can gather opinions and reach a consensus. The trainers team at EUM will also start providing more inputs as they put their hands on more data.

I would say that as long as FCI is pre-operational, we are still quite free to change things (radically) without worrying too much. After it's operational we can think of a warning system for less-established RGBs, although the process for fine-tuning could go on for years :)

[pnuu]

For FCI IR Sandwich I've used vis_06 for the visible portion, works pretty nice.

[gerritholl]

Inevitably, FCI composites would differ from SEVIRI composites that use HRV. We would need to wait for a community consensus on how those definitions will be. I don't agree that we need to entirely forget about those composites, though; new composites can be developed that include comparable information at a 500-metre spatial resolution level. There's nothing that stops us with Pytroll/Satpy on proposing possible ways forward and marking those ways as experimental.

Yes, for new FCI composites we can change everything all the time, but with the analysis I noticed that even for SEVIRI we don't always match community definitions, and that's a lot more dicey to change.

[pnuu]

In spite of my skepticism against working on HRV composites for FCI, I did a test:

And the quick'n'dirty implementation:

#!/usr/bin/env python

import glob

import hdf5plugin
from satpy.writers import to_image
from satpy import Scene
from satpy.composites import GenericCompositor

fnames = glob.glob("/home/lahtinep/data/satellite/geo/fci_2022_compressed/*_0073_*.nc")
scn = Scene(reader='fci_l1c_nc', filenames=fnames)
scn.load(['vis_05', 'vis_06', 'vis_08', 'nir_16'])
scn2 = scn.resample('EPSG_3035_2km', resampler='gradient_search')
hrv = (scn2['vis_05'] + scn2['vis_06'] + scn2['vis_08']) / 3.0
hrv.attrs['area'] = scn2['nir_16'].attrs['area']
comp = GenericCompositor('hrv_fog')
res = comp((scn2['nir_16'], hrv, hrv))
img = to_image(res)
img.stretch()
img.save('/tmp/test.png')

[pnuu]

Also, just using vis06 instead of HRV isn't half bad, maybe a bit too red:

[gerritholl]

Note on the IR sandwich: here we might want to resist the EUMeTrain standard and convince them to change. The EUMeTrain sandwhich colorises the colder part of the range using the rainbow colormap, whereas satpy applies spectral. We might want to bring this up with EUMeTrain. I'm not sure everyone is even aware satpy is doing something different.

[ameraner]

After #2780, the main missing point to close this issue are the missing tropical recipes for some RGBs. Considering that the current recipes basically cover the midlatitude cases, we could either

• add the tropical ones with the suffix _tropical, i.e. to end up with night_microphysics and night_microphysics_tropical
• introduce the suffix midlat and slowly deprecate the old ones, i.e. to end up with night_microphysics_midlat and night_microphysics_tropical

I would probably vote for the second since it's the clearest long-term, even though it introduces some duplication until we have a safe mechanism to deprecate composite (names)..

[gerritholl]

For naming tropical or midlatitude variants, we might want to find a solution to https://github.com/pytroll/satpy/issues/2644 first.

[ameraner]

I agree that the changes proposed in #2644 are very useful, but they would also require a new way of querying composites for users, which will take time to implement and to gain acceptance... I would be in favour of completing the list by adding the tropical recipes to the mix now, and then hopefully have a better query mechanism soon to improve the overall situation.

[gerritholl]

I agree for the tropical recipes. Not sure about adding _midlat versions specifically, though. Maybe we can raise that question at the Pytroll User Days.