Let the radiances be negative across all readers

[mraspaud]

Feature Request

At the moment, different readers have different behaviours when it comes to clipping or not negative radiances. As this has been pointed out by @simonrp84, it can be desirable to keep the negative radiances for eg sensor noise estimation. Even though different space agency probably recommend different behaviours, it is important that satpy stays consistent in this I think.

This issue is raised to have a place to discuss different aspects of this standardization.

Things to take into consideration:

1. Should there be clipping before calibration to Reflectances and Brightness Temperatures? as @gerritholl points out, negative radiances generate negative reflectances, which could be accepted, and negative radiances generated invalid (NaN) brightness temperatures. Both could be interesting for the users.
2. The removal of the clipping in the readers implies that the compositing should take care of that instead. This is not difficult, but should not be missed.
3. As a more general point, this shows that scientific users might want to have readers alter the data as little as possible before calibration or corrections are applied, and we need to make the API more flexible to allow them have more control over the different steps involved.
4. Documentation for users and guidelines for developers should be written/updated to make sure everyone understands what the standard in satpy is, and red flags should be raised for readers not following the standard.

There are certainly more to think about here, please do comment.

[djhoese]

Is there anything mathematically that requires that composites handle this? I think typically this is handled "automatically" by enhancements. The one compositor I can think of that these negatives may play a part is my "RatioSharpeningCompositor" stuff where I take the ratio of two bands (ex. red / low_res_red) and then multiply the other bands by that ratio. I've updated this compositor in the past to make sure the ratio is never negative so removing the clipping from the readers shouldn't be a problem there.

[gerritholl]

Mathematically no, but if one band out of three is masked (NaN brightness temperatures) the impact on the colour is much larger than when it's clipped to the lower end of a range.

[pnuu]

Mathematically no, but if one band out of three is masked (NaN brightness temperatures) the impact on the colour is much larger than when it's clipped to the lower end of a range.

Only if linear stretch is used, but yeah, some users (and built-in composites) do that.

[djhoese]

The GenericCompositor will "share" the mask between any input bands so you make a very good point @gerritholl. If a negative radiance produces a NaN temperature and is then included in a RGB then it would effect all pixels of the RGB. Maybe it should be up to the IR calibration in a reader to clip the radiances to 0? Hhhmmm but that's still a "esthetic" change rather than a "scientific" requirement.

[mraspaud]

log-based enhancements won't like this (eg cira-stretch, reinhardt).

[djhoese]

@mraspaud They already have to deal with negatives for other readers. And the enhancements are the component that should be dealing with this.

[mraspaud]

ok. We just have to check that we don't break anything when removing the clipping.

[sfinkens]

Good point, thanks for raising this! Would clipping remain the default behaviour? Your third point reads like a proposal for generic calibration algorithms in Satpy that users can choose from. That would be great! :slightly_smiling_face:

[simonrp84]

If radiances can be clipped by the enhancements / composites then I'd strongly push for the default to be no clipping: There's a big risk that research users won't know about this behaviour and will just use the default.

[sfinkens]

That makes sense, they'll probably notice the negative values and if they don't want them they can enable clipping.

[djhoese]

I personally feel like there should be no option for clipping in the readers unless it is scientifically useful/meaningful. Otherwise it is more of an "esthetic" change and should be handled in the RGB composites or enhancements.

[simonrp84]

I had a quick search through the code and it seems that the following readers do some manner of clipping:

aaap_l1b (line 548):

• Clips negative VIS channel (reflectance?) to zero.
• No clipping for IR channels.

agri_l1 (line 63):

• Clips negative VIS reflectances to zero.
• No clipping for IR channels.

ahi_hsd:

• Clips negative VIS + IR radiances to zero.
• Clips negative VIS reflectances to zero.
• Clips negative IR temperatures to zero.

goes_imager_hrit (line 439):

• Clips negative VIS reflectances to zero.
• Clips negative IR temperatures to zero.

goes_imager_nc (lines 851, 896, 923):

• Clips negative VIS radiances to zero.
• Clips negative IR radiances to zero.
• Clips negative VIS reflectances to zero.

sar_c_safe (line 599):

• Clips negative data to zero (is this an issue? Don't know enough about SAR).

seviri_base (line 490):

• Clips negative VIS radiances to None (I think, maybe clips to zero)
• Clips negative IR radiances to None (I think, maybe clips to zero)

viirs_compact (line 280):

• Clips negative VIS radiances to zero.
• Clips negative IR radiances to zero.
• Clips negative VIS reflectances to zero.
• Clips negative IR temperatures to zero.

[mraspaud]

Thanks for the investigation work @simonrp84 !

[sfinkens]

Just catched up with the conversation on slack where @simonrp84 mentioned that negative radiances can't be used for anything physical. So I think it would make sense to keep the clipping option. This would be convenient for scientific users who are processing the data in their algorithms.

[ameraner]

See also https://github.com/pytroll/satpy/pull/2499 where optional radiance clipping through a config key was implemented by @ghiggi (currently only for abi_l1b)