Spectral Band Adjustment Factor (SBAF) feature

[pdebuyl]

Feature Request

Is your feature request related to a problem? Please describe.

For transferring retrieval or color compositions between instruments, it can be useful to map one sensor to another. For IR imagers, the technique is called "Spectral Band Adjustment Factor" (SBAF).

Describe the solution you'd like

I would like the feature to be able to select and apply a set of parameters for applying SBAF to a geostationary imager.

Describe any changes to existing user workflow

The feature would be optional, so not breaking backwards compatibility, and can be implemented within satpy (or pyspectral if more appropriate).

Additional context

The current modifiers seem to act mostly on a single channel at a time. For creating a dust RGB for instance, I don't think I could do something like

  dust:
    modifier: !!python/name:satpy.modifiers.spectral.SBAF
    compositor: !!python/name:satpy.composites.core.GenericCompositor
    prerequisites:
    - compositor: !!python/name:satpy.composites.arithmetic.DifferenceCompositor
      prerequisites:
      - 12.0
      - 10.8
    - compositor: !!python/name:satpy.composites.arithmetic.DifferenceCompositor
      prerequisites:
      - 10.8
      - 8.7
    - 10.8
    standard_name: dust

so that the individual bands would be SBAF corrected before being fed to the compositors.

[gerritholl]

I think you could do

modifiers:
  sbaf_corrected:
  modifier: !!python/name:satpy.modifiers.spectral.SBAF
  prerequisites:
    - 8.7
    - 10.8
    - 12.0
    - 13.4
  
composites:  
  dust:
    compositor: !!python/name:satpy.composites.core.GenericCompositor
    prerequisites:
    - compositor: !!python/name:satpy.composites.arithmetic.DifferenceCompositor
      prerequisites:
      - 12.0
        modifiers: [sbaf_corrected]
      - 10.8
        modifiers: [sbaf_corrected]
    - compositor: !!python/name:satpy.composites.arithmetic.DifferenceCompositor
      prerequisites:
      - 10.8
        modifiers: [sbaf_corrected]
      - 8.7
        modifiers: [sbaf_corrected]
    - 10.8
      modifiers: [sbaf_corrected]
    standard_name: dust

but it would call SBAF three times rather than once. Or maybe I am misunderstanding something?

[mraspaud]

with enough caching, calling the modifier three times is probably not an issue indeed

[pdebuyl]

How do I control the caching? In some cases, I need all IR bands (WV_062, WV_073, IR_087, IR_097, IR_108, IR_120, IR_134 for SEVIRI for instance) to be kept.

[djhoese]

As @gerritholl mentioned on slack, if the created dask array has the same ".name" (NOTE: this is not the xarray DataArray, but the underlying dask array) then dask should realize it is the same computation and only compute it once. So if you had a DataArray you could do print(data_arr.data.name) and compare between executions.

[pdebuyl]

Update: after doing some basic research and prototyping I had to face an annoying issue. There is no de facto standard for SBAF definition.

For a first test, I ended up relying on NASA SatCORPS's SBAF tool (the IASI-based one): https://satcorps.larc.nasa.gov/cgi-bin/site/showdoc?mnemonic=SBAF&mode=IR

The web tool is online since some time and is probably used more widely than other options. Fortunately, the principle is simple: it works band by band by defining the output as a polynomial of the input. Calling T_in the sensor brightness temperature and T_out the adjusted one, we have T_out = c_0 + c_1 T + c_2 T^2 + .... The web tool proposes orders 1, 2, and 3. I ran the tool for bands 8.5, 10.5, and 12.3 µm (approx) for GOES-16 ABI to MTG FCI (using region MSG).

modifiers:
  sbaf_corrected:
    modifier: !!python/name:satpy.modifiers.spectral.SBAFSingle
    coefficients_file: "/path/to/sbaf/goes_fci_ash_bands.json"

composites:

  ash_sbaf:
    compositor: !!python/name:satpy.composites.core.GenericCompositor
    prerequisites:
    - compositor: !!python/name:satpy.composites.arithmetic.DifferenceCompositor
      prerequisites:
      - name: ir_123
        modifiers: [sbaf_corrected]
      - name: ir_105
        modifiers: [sbaf_corrected]
    - compositor: !!python/name:satpy.composites.arithmetic.DifferenceCompositor
      prerequisites:
      - name: ir_105
        modifiers: [sbaf_corrected]
      - name: ir_87
        modifiers: [sbaf_corrected]
    - name: ir_105
      modifiers: [sbaf_corrected]
    standard_name: ash_sbaf

Implementation:

class SBAFSingle(ModifierBase):
    """Appy Spectral Band Adjustment Factor per-band

    Correct the band individucally according to the polynomial coefficients passed in the
    file `coefficients_file`. The file is a json dict where the entries are named as the
    sensor bands. The coefficients of a band are given as a list, starting at order 0 (at
    least two coefficients are needed), using brightness temperature (BT) as units.
    """

    def __call__(self, projectables, optional_datasets=None, **info):
        """Apply the SBAF correction to a single band."""
        band = projectables[0]
        coef_file = self.attrs.get("coefficients_file")
        with open(coef_file, 'r') as f:
            coefs = json.load(f)
        # Keep only a single band
        coefs = coefs[band.attrs['name']]
        assert len(coefs)>1
        logger.info("Applying single-band SBAF correction")

        # Apply polynomial fit
        data = coefs[1]*band.data + coefs[0]
        for i, coef in enumerate(coefs[2:]):
            data += coef*band.data**(i+2)
        res = xr.DataArray(data, dims=band.dims, attrs=band.attrs, coords=band.coords)

        return res

Notes:

• I didn't have the choice of units as modifiers receive the calibrated input (so BT for thermal bands).
• I special-cased the order 1 because band.data**0 is not optimal.
• I used a json file to store the parameters, could I put them in the yaml instead? This is rather custom so probably any user would put a dedicated parametrization in the yaml ?

Now: I started a "N-band" version where one gives all thermal bands as prerequisites to the modifier and starting the call by self_band, *ir_bands = projectables so that self_band gives the name of the band and ir_bands is the list of all thermal bands on which I can apply multi-band SBAF.

[djhoese]

How big is the contents of the JSON file?

I didn't have the choice of units as modifiers receive the calibrated input (so BT for thermal bands).

What do you need/want? Radiance?

[pdebuyl]

The JSON file is very small here. For order 3, 4 numbers per band for instance. (c_0 to c_order).

[pdebuyl]

Regarding the units, the comment was also because I realized that the modifier is executed after calibration and that one needs to be aware of if. For thermal bands, the data is in BT, for visible bands in reflectance. Nothing bad, but it wasn't obvious to me when first trying to work with modifiers.