satpy icon indicating copy to clipboard operation
satpy copied to clipboard

Add double alpha channel support and improve metadata behaviours for BackgroundCompositor

Open yukaribbba opened this issue 2 years ago • 49 comments

Current BackgroundCompositor assumes that only foreground has an alpha band. What if both of them do? Like the newly added #2557 , both LowCloudCompositor and HighCloudCompositor generate images with alpha channels.

Here's an example of a high_cloud stacked on a low_cloud: This is the result of current compositor. Apparently it can't handle the background alpha correctly. image

Below is by this PR. It recognizes both alpha channels, and builds a new alpha. image

  • [x] Closes #2695
  • [x] Tests added
  • [x] Fully documented

yukaribbba avatar Dec 16 '23 05:12 yukaribbba

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 95.94%. Comparing base (f5a8532) to head (05fdcbf). Report is 1 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #2696   +/-   ##
=======================================
  Coverage   95.94%   95.94%           
=======================================
  Files         379      379           
  Lines       53673    53693   +20     
=======================================
+ Hits        51494    51515   +21     
+ Misses       2179     2178    -1     
Flag Coverage Δ
behaviourtests 4.09% <8.57%> (+<0.01%) :arrow_up:
unittests 96.04% <100.00%> (+<0.01%) :arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

codecov[bot] avatar Dec 16 '23 08:12 codecov[bot]

Pull Request Test Coverage Report for Build 8762065249

Details

  • 0 of 0 changed or added relevant lines in 0 files are covered.
  • No unchanged relevant lines lost coverage.
  • Overall coverage remained the same at 0.0%

Totals Coverage Status
Change from base Build 8732515933: 0.0%
Covered Lines: 0
Relevant Lines: 0

💛 - Coveralls

coveralls avatar Dec 16 '23 10:12 coveralls

Could you describe the use case here? If the foreground has invalid pixels that you don't want to be filled in with background, why are you using the background compositor? Additionally, I believe there are "Filler" compositors that will fill in invalid data with a specific value. This could be done before calling the BackgroundCompositor.

djhoese avatar Dec 16 '23 15:12 djhoese

Well this is a little more complicated than I expected. I have to organize my thoughts. I think maybe I need a third channel used for masking area, just like there's a "high_resolution_band" in SharpenedCompositor

yukaribbba avatar Dec 16 '23 15:12 yukaribbba

pre-commit.ci autofix

yukaribbba avatar Dec 18 '23 08:12 yukaribbba

@djhoese Ready for review :}

yukaribbba avatar Dec 19 '23 11:12 yukaribbba

In your updated description, what is currently produced in main if both inputs have alphas? What is this PR choosing to do (sorry, it isn't clear to me from the picture)?

djhoese avatar Dec 19 '23 15:12 djhoese

Here's an example of a high_cloud stacked on a low_cloud: This is the result of current compositor. Apparently it can't handle the background alpha correctly. image

Below is by this PR. It recognizes both alpha channels, and generates a new alpha channel. image

yukaribbba avatar Dec 19 '23 16:12 yukaribbba

Just a thought, but could the first use-case be implemented by combining the existing Background and Masking compositors?

pnuu avatar Dec 19 '23 16:12 pnuu

Just a thought, but could the first use-case be implemented by combining the existing Background and Masking compositors?

Well maybe, but that looks a little complicated and I want to do it in one step....

yukaribbba avatar Dec 19 '23 16:12 yukaribbba

So is another way of summarizing this PR that use case 1 is a new feature and use case 2 (the two alphas) is a bug fix?

I think overall I agree with @pnuu. There is something to be said about individual components having a single purpose (the unix philosophy). Especially when that purpose (masking) seems to be covered by an existing component. Even if it requires using two or more compositors this could be done with "inline" compositors in a single definition if necessary.

djhoese avatar Dec 19 '23 17:12 djhoese

Understood. So I decide to remove the first part, keeping it for myself. If someone wants that in the future I'll re-add it. I'll still commit one with fixes you mentioned as a backup here.

yukaribbba avatar Dec 20 '23 01:12 yukaribbba

The commit 707334f is the last one with masking function.

yukaribbba avatar Dec 20 '23 01:12 yukaribbba

There's a problem that happens both on main and this PR. If we stack L/L or RGB/RGB together(well that's boring), we'll get an error.

My test script:

files = find_files_and_readers(base_dir='C:/Users/45107/Downloads/Sat/Geo/gk2a_20231215_1600_FLDK', reader='ami_l1b')

area = AreaDefinition(area_id='world', description='world latlon', proj_id='latlon',
                      projection='+proj=latlon +datum=WGS84 +ellps=WGS84',
                      width=4096, height=2048, area_extent=(-180, -90, 180, 90))

scn4 = Scene(filenames=files)
product = 'blackmarble_with_blackmarble' # RGB/RGB
scn4.load([product])
scn4 = scn4.resample(area, resampler='nearest') # Whether resampled or not, error raises
scn4.show(product)
  static_night:
    compositor: !!python/name:satpy.composites.StaticImageCompositor    
    filename: C:/Users/45107/Downloads/Sat/Geo/satpy_night.tif
    standard_name: static_image_spec

  blackmarble_with_blackmarble:
    compositor: !!python/name:satpy.composites.BackgroundCompositor
    prerequisites:
      - name: static_night
      - name: static_night

DEBUG

[DEBUG: 2023-12-20 10:54:54 : satpy.readers.yaml_reader] Reading ('C:\\Users\\45107\\miniforge3\\Lib\\site-packages\\satpy\\etc\\readers\\ami_l1b.yaml',)
[DEBUG: 2023-12-20 10:54:54 : satpy.readers.yaml_reader] Reading ('C:\\Users\\45107\\miniforge3\\Lib\\site-packages\\satpy\\etc\\readers\\ami_l1b.yaml',)
[DEBUG: 2023-12-20 10:54:54 : satpy.readers.yaml_reader] Assigning to ami_l1b: ['C:/Users/45107/Downloads/Sat/Geo/gk2a_20231215_1600_FLDK\\gk2a_ami_le1b_ir105_fd020ge_202312151600.nc', 'C:/Users/45107/Downloads/Sat/Geo/gk2a_20231215_1600_FLDK\\gk2a_ami_le1b_sw038_fd020ge_202312151600.nc']
[DEBUG: 2023-12-20 10:54:54 : rasterio.env] PROJ data files are available at built-in paths.
C:\Users\45107\miniforge3\Lib\site-packages\xarray\core\dataset.py:278: UserWarning: The specified chunks separate the stored chunks along dimension "dim_image_y" starting at index 4096. This could degrade performance. Instead, consider rechunking after loading.
  warnings.warn(
C:\Users\45107\miniforge3\Lib\site-packages\xarray\core\dataset.py:278: UserWarning: The specified chunks separate the stored chunks along dimension "dim_image_x" starting at index 4096. This could degrade performance. Instead, consider rechunking after loading.
  warnings.warn(
[DEBUG: 2023-12-20 10:54:54 : satpy.composites.config_loader] Looking for composites config file ami.yaml
[DEBUG: 2023-12-20 10:54:54 : satpy.composites.config_loader] Looking for composites config file visir.yaml
[DEBUG: 2023-12-20 10:54:54 : pyorbital.tlefile] Path to the Pyorbital configuration (where e.g. platforms.txt is found): C:\Users\45107\miniforge3\Lib\site-packages\pyorbital\etc
[DEBUG: 2023-12-20 10:54:54 : satpy.readers.yaml_reader] Reading ('C:\\Users\\45107\\miniforge3\\Lib\\site-packages\\satpy\\etc\\readers\\generic_image.yaml',)
[DEBUG: 2023-12-20 10:54:54 : satpy.readers.yaml_reader] Assigning to generic_image: [<FSFile "C:/Users/45107/Downloads/Sat/Geo/satpy_night.tif">]
[DEBUG: 2023-12-20 10:54:54 : rasterio.env] Entering env context: <rasterio.env.Env object at 0x000001E29B666390>
[DEBUG: 2023-12-20 10:54:54 : rasterio.env] Starting outermost env
[DEBUG: 2023-12-20 10:54:54 : rasterio.env] No GDAL environment exists
[DEBUG: 2023-12-20 10:54:54 : rasterio.env] New GDAL environment <rasterio._env.GDALEnv object at 0x000001E29D5F69E0> created
[DEBUG: 2023-12-20 10:54:54 : rasterio._filepath] Installing FilePath filesystem handler plugin...
[DEBUG: 2023-12-20 10:54:54 : rasterio._env] GDAL_DATA found in environment.
[DEBUG: 2023-12-20 10:54:54 : rasterio._env] PROJ data files are available at built-in paths.
[DEBUG: 2023-12-20 10:54:54 : rasterio._env] Started GDALEnv: self=<rasterio._env.GDALEnv object at 0x000001E29D5F69E0>.
[DEBUG: 2023-12-20 10:54:54 : rasterio.env] Entered env context: <rasterio.env.Env object at 0x000001E29B666390>
[DEBUG: 2023-12-20 10:54:54 : rasterio._base] Sharing flag: 0
[DEBUG: 2023-12-20 10:54:54 : rasterio._base] Nodata success: 0, Nodata value: 0.000000
[DEBUG: 2023-12-20 10:54:54 : rasterio._base] Nodata success: 0, Nodata value: 0.000000
[DEBUG: 2023-12-20 10:54:54 : rasterio._base] Nodata success: 0, Nodata value: 0.000000
[DEBUG: 2023-12-20 10:54:54 : rasterio._base] Dataset <open DatasetReader name='C:/Users/45107/Downloads/Sat/Geo/satpy_night.tif' mode='r'> is started.
[DEBUG: 2023-12-20 10:54:54 : rasterio.env] Exiting env context: <rasterio.env.Env object at 0x000001E29B666390>
[DEBUG: 2023-12-20 10:54:54 : rasterio.env] Cleared existing <rasterio._env.GDALEnv object at 0x000001E29D5F69E0> options
[DEBUG: 2023-12-20 10:54:54 : rasterio._env] Stopped GDALEnv <rasterio._env.GDALEnv object at 0x000001E29D5F69E0>.
[DEBUG: 2023-12-20 10:54:54 : rasterio.env] Exiting outermost env
[DEBUG: 2023-12-20 10:54:54 : rasterio.env] Exited env context: <rasterio.env.Env object at 0x000001E29B666390>
[DEBUG: 2023-12-20 10:54:54 : rasterio.env] Entering env context: <rasterio.env.Env object at 0x000001E29D1C8B90>
[DEBUG: 2023-12-20 10:54:54 : rasterio.env] Starting outermost env
[DEBUG: 2023-12-20 10:54:54 : rasterio.env] No GDAL environment exists
[DEBUG: 2023-12-20 10:54:54 : rasterio.env] New GDAL environment <rasterio._env.GDALEnv object at 0x000001E29D5F7B20> created
[DEBUG: 2023-12-20 10:54:54 : rasterio._env] GDAL_DATA found in environment.
[DEBUG: 2023-12-20 10:54:54 : rasterio._env] PROJ data files are available at built-in paths.
[DEBUG: 2023-12-20 10:54:54 : rasterio._env] Started GDALEnv: self=<rasterio._env.GDALEnv object at 0x000001E29D5F7B20>.
[DEBUG: 2023-12-20 10:54:54 : rasterio.env] Entered env context: <rasterio.env.Env object at 0x000001E29D1C8B90>
[DEBUG: 2023-12-20 10:54:54 : rasterio._base] Sharing flag: 0
[DEBUG: 2023-12-20 10:54:54 : rasterio._base] Nodata success: 0, Nodata value: 0.000000
[DEBUG: 2023-12-20 10:54:54 : rasterio._base] Nodata success: 0, Nodata value: 0.000000
[DEBUG: 2023-12-20 10:54:54 : rasterio._base] Nodata success: 0, Nodata value: 0.000000
[DEBUG: 2023-12-20 10:54:54 : rasterio._base] Dataset <open DatasetReader name='C:/Users/45107/Downloads/Sat/Geo/satpy_night.tif' mode='r'> is started.
[DEBUG: 2023-12-20 10:54:54 : rasterio.env] Exiting env context: <rasterio.env.Env object at 0x000001E29D1C8B90>
[DEBUG: 2023-12-20 10:54:54 : rasterio.env] Cleared existing <rasterio._env.GDALEnv object at 0x000001E29D5F7B20> options
[DEBUG: 2023-12-20 10:54:54 : rasterio._env] Stopped GDALEnv <rasterio._env.GDALEnv object at 0x000001E29D5F7B20>.
[DEBUG: 2023-12-20 10:54:54 : rasterio.env] Exiting outermost env
[DEBUG: 2023-12-20 10:54:54 : rasterio.env] Exited env context: <rasterio.env.Env object at 0x000001E29D1C8B90>
[DEBUG: 2023-12-20 10:54:54 : satpy.composites.config_loader] Looking for composites config file images.yaml
[DEBUG: 2023-12-20 10:54:54 : satpy.composites.config_loader] No composite config found called images.yaml
[DEBUG: 2023-12-20 10:54:54 : satpy.readers.generic_image] Reading 'image.'
[WARNING: 2023-12-20 10:54:54 : satpy.readers.generic_image] cannot convert float NaN to integer
[DEBUG: 2023-12-20 10:54:54 : satpy.writers] Adding enhancement configuration from file: C:\Users\45107\miniforge3\Lib\site-packages\satpy\etc\enhancements\generic.yaml
[DEBUG: 2023-12-20 10:54:55 : satpy.writers] Adding enhancement configuration from file: D:\satpy_config\enhancements\generic.yaml
[DEBUG: 2023-12-20 10:54:55 : satpy.writers] Data for DataID(name='static_night') will be enhanced with options:
	[{'name': 'stretch', 'method': <function stretch at 0x000001E29D632200>, 'kwargs': {'stretch': 'crude', 'min_stretch': [0, 0, 0], 'max_stretch': [255, 255, 255]}}]
[DEBUG: 2023-12-20 10:54:55 : trollimage.xrimage] Applying stretch crude with parameters {'min_stretch': [0, 0, 0], 'max_stretch': [255, 255, 255]}
[DEBUG: 2023-12-20 10:54:55 : satpy.writers] Adding enhancement configuration from file: C:\Users\45107\miniforge3\Lib\site-packages\satpy\etc\enhancements\generic.yaml
[DEBUG: 2023-12-20 10:54:55 : satpy.writers] Adding enhancement configuration from file: D:\satpy_config\enhancements\generic.yaml
[DEBUG: 2023-12-20 10:54:55 : satpy.writers] Data for DataID(name='static_night') will be enhanced with options:
	[{'name': 'stretch', 'method': <function stretch at 0x000001E29D632200>, 'kwargs': {'stretch': 'crude', 'min_stretch': [0, 0, 0], 'max_stretch': [255, 255, 255]}}]
[DEBUG: 2023-12-20 10:54:55 : trollimage.xrimage] Applying stretch crude with parameters {'min_stretch': [0, 0, 0], 'max_stretch': [255, 255, 255]}
[DEBUG: 2023-12-20 10:54:55 : satpy.scene] Resampling DataID(name='static_night')
[INFO: 2023-12-20 10:54:55 : satpy.scene] Not reducing data before resampling.
[DEBUG: 2023-12-20 10:54:55 : satpy.resample] Computing kd-tree parameters
[DEBUG: 2023-12-20 10:54:55 : satpy.resample] Resampling where-0c2efa9774b75cb3d825d9cc0911c8b2
Traceback (most recent call last):
  File "C:\Users\45107\miniforge3\Lib\site-packages\satpy\dataset\data_dict.py", line 169, in __getitem__
    return super(DatasetDict, self).__getitem__(item)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
KeyError: 'blackmarble_with_blackmarble'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\45107\PycharmProjects\Satellite\bcdef.py", line 16, in <module>
    scn4.show(product)
  File "C:\Users\45107\miniforge3\Lib\site-packages\satpy\scene.py", line 1010, in show
    img = get_enhanced_image(self[dataset_id].squeeze(), overlay=overlay)
                             ~~~~^^^^^^^^^^^^
  File "C:\Users\45107\miniforge3\Lib\site-packages\satpy\scene.py", line 825, in __getitem__
    return self._datasets[key]
           ~~~~~~~~~~~~~~^^^^^
  File "C:\Users\45107\miniforge3\Lib\site-packages\satpy\dataset\data_dict.py", line 171, in __getitem__
    key = self.get_key(item)
          ^^^^^^^^^^^^^^^^^^
  File "C:\Users\45107\miniforge3\Lib\site-packages\satpy\dataset\data_dict.py", line 158, in get_key
    return get_key(match_key, self.keys(), num_results=num_results,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\45107\miniforge3\Lib\site-packages\satpy\dataset\data_dict.py", line 107, in get_key
    raise KeyError("No dataset matching '{}' found".format(str(key)))
KeyError: "No dataset matching 'DataQuery(name='blackmarble_with_blackmarble')' found"

Do you think it's related to #2668 and #2690 ?

yukaribbba avatar Dec 20 '23 02:12 yukaribbba

scn4 = scn4.resample

First, don't reassign the Scene variable. Instead use a new name to avoid Python garbage collecting the initial scn4.

The error is a KeyError because the composite never successfully generated a result. From the log output it isn't clear to me what happened. Some exception is being silenced/hidden.

djhoese avatar Dec 20 '23 15:12 djhoese

Ok but this also happens when there's no resampling.

yukaribbba avatar Dec 20 '23 15:12 yukaribbba

Right, but the compositor is called in both cases. I was just pointing out that it is not a good idea to reassign the Scene. I realize this isn't the primary issue, but just a heads up to avoid it.

My only other guess is that the dependency tree is getting confused because you're depending on a single dataset twice in your composite but I don't think that should be a problem.

You could put a huge try/except around everything after match_data_arrays in the compositor and see what error(s) you get.

djhoese avatar Dec 20 '23 15:12 djhoese

Didn't get anything from try/except. But you're right, this may not be a problem since we rarely do that...

yukaribbba avatar Dec 20 '23 15:12 yukaribbba

If you put print statements in the compositor, does it get called? Does it get all the way to returning a DataArray?

djhoese avatar Dec 20 '23 15:12 djhoese

print data

[<xarray.DataArray (y: 2048, x: 4096)>
dask.array<where, shape=(2048, 4096), dtype=float32, chunksize=(2048, 4096), chunktype=numpy.ndarray>
Coordinates:
  * x            (x) float64 -180.0 -179.9 -179.8 -179.7 ... 179.8 179.9 180.0
  * y            (y) float64 89.96 89.87 89.78 89.69 ... -89.78 -89.87 -89.96
    spatial_ref  int32 0
    crs          object GEOGCRS["WGS 84",DATUM["World Geodetic System 1984",E...
    bands        <U1 'A', <xarray.DataArray (y: 2048, x: 4096)>
dask.array<where, shape=(2048, 4096), dtype=float32, chunksize=(2048, 4096), chunktype=numpy.ndarray>
Coordinates:
  * x            (x) float64 -180.0 -179.9 -179.8 -179.7 ... 179.8 179.9 180.0
  * y            (y) float64 89.96 89.87 89.78 89.69 ... -89.78 -89.87 -89.96
    spatial_ref  int32 0
    crs          object GEOGCRS["WGS 84",DATUM["World Geodetic System 1984",E...
    bands        <U1 'A', <xarray.DataArray (y: 2048, x: 4096)>
dask.array<where, shape=(2048, 4096), dtype=float32, chunksize=(2048, 4096), chunktype=numpy.ndarray>
Coordinates:
  * x            (x) float64 -180.0 -179.9 -179.8 -179.7 ... 179.8 179.9 180.0
  * y            (y) float64 89.96 89.87 89.78 89.69 ... -89.78 -89.87 -89.96
    spatial_ref  int32 0
    crs          object GEOGCRS["WGS 84",DATUM["World Geodetic System 1984",E...
    bands        <U1 'A']

print res

<xarray.DataArray 'where-7cb3c772602cf5919c594884bafe9c7a' (bands: 3, y: 2048,
                                                            x: 4096)>
dask.array<where, shape=(3, 2048, 4096), dtype=float32, chunksize=(1, 2048, 4096), chunktype=numpy.ndarray>
Coordinates:
  * x            (x) float64 -180.0 -179.9 -179.8 -179.7 ... 179.8 179.9 180.0
  * y            (y) float64 89.96 89.87 89.78 89.69 ... -89.78 -89.87 -89.96
    spatial_ref  int32 0
    crs          object GEOGCRS["WGS 84",DATUM["World Geodetic System 1984",E...
  * bands        (bands) <U1 'R' 'G' 'B'
Attributes:
    wavelength:              None
    name:                    blackmarble_with_blackmarble
    _satpy_id:               DataID(name='blackmarble_with_blackmarble')
    prerequisites:           [DataQuery(name='static_night'), DataQuery(name=...
    optional_prerequisites:  []
    sensor:                  None
    mode:                    RGB

yukaribbba avatar Dec 20 '23 15:12 yukaribbba

Looks like the compositor did its job correctly. The error is somewhere else.

yukaribbba avatar Dec 20 '23 16:12 yukaribbba

Does this fail to load if you use main instead of this branch?

djhoese avatar Dec 20 '23 16:12 djhoese

It also happens.

print data

[<xarray.DataArray (y: 2048, x: 4096)>
dask.array<getitem, shape=(2048, 4096), dtype=float32, chunksize=(2048, 4096), chunktype=numpy.ndarray>
Coordinates:
  * x            (x) float64 -180.0 -179.9 -179.8 -179.7 ... 179.8 179.9 180.0
  * y            (y) float64 89.96 89.87 89.78 89.69 ... -89.78 -89.87 -89.96
    spatial_ref  int32 0
    bands        <U1 'R'
    crs          object GEOGCRS["WGS 84",DATUM["World Geodetic System 1984",E..., <xarray.DataArray (y: 2048, x: 4096)>
dask.array<getitem, shape=(2048, 4096), dtype=float32, chunksize=(2048, 4096), chunktype=numpy.ndarray>
Coordinates:
  * x            (x) float64 -180.0 -179.9 -179.8 -179.7 ... 179.8 179.9 180.0
  * y            (y) float64 89.96 89.87 89.78 89.69 ... -89.78 -89.87 -89.96
    spatial_ref  int32 0
    bands        <U1 'G'
    crs          object GEOGCRS["WGS 84",DATUM["World Geodetic System 1984",E..., <xarray.DataArray (y: 2048, x: 4096)>
dask.array<getitem, shape=(2048, 4096), dtype=float32, chunksize=(2048, 4096), chunktype=numpy.ndarray>
Coordinates:
  * x            (x) float64 -180.0 -179.9 -179.8 -179.7 ... 179.8 179.9 180.0
  * y            (y) float64 89.96 89.87 89.78 89.69 ... -89.78 -89.87 -89.96
    spatial_ref  int32 0
    bands        <U1 'B'
    crs          object GEOGCRS["WGS 84",DATUM["World Geodetic System 1984",E...]

print res

<xarray.DataArray 'where-a7023d79eee34bda86d5737ee7a3d36e' (bands: 3, y: 2048,
                                                            x: 4096)>
dask.array<where, shape=(3, 2048, 4096), dtype=float32, chunksize=(1, 2048, 4096), chunktype=numpy.ndarray>
Coordinates:
  * x            (x) float64 -180.0 -179.9 -179.8 -179.7 ... 179.8 179.9 180.0
  * y            (y) float64 89.96 89.87 89.78 89.69 ... -89.78 -89.87 -89.96
    spatial_ref  int32 0
    crs          object GEOGCRS["WGS 84",DATUM["World Geodetic System 1984",E...
  * bands        (bands) <U1 'R' 'G' 'B'
Attributes:
    wavelength:              None
    name:                    blackmarble_with_blackmarble
    _satpy_id:               DataID(name='blackmarble_with_blackmarble')
    prerequisites:           [DataQuery(name='static_night'), DataQuery(name=...
    optional_prerequisites:  []
    sensor:                  None
    mode:                    RGB

yukaribbba avatar Dec 20 '23 16:12 yukaribbba

Still keyerror

yukaribbba avatar Dec 20 '23 16:12 yukaribbba

My only other guess is that the dependency tree is getting confused because you're depending on a single dataset twice in your composite

But we can do this without any problems.

  IR105_RGB:
    compositor: !!python/name:satpy.composites.GenericCompositor
    prerequisites:
      - name: IR105
      - name: IR105
      - name: IR105

yukaribbba avatar Dec 20 '23 16:12 yukaribbba

In the compositor code can you take the result and set a value for .attrs["sensor"] = "viirs" and .attrs["start_time"] = datetime.utcnow() and see if it behaves better? Oh you could also try adding a standard_name to the composite YAML for the background compositor. Technically for an actual output image you'd want a corresponding enhancement for that standard_name, but if you're not concerned with what the final output looks like then that doesn't matter.

djhoese avatar Dec 20 '23 16:12 djhoese

Sorry it's late at night in my time and I have to go to bed. I'll try what you said tomorrow.

yukaribbba avatar Dec 20 '23 16:12 yukaribbba

In the compositor code can you take the result and set a value for .attrs["sensor"] = "viirs" and .attrs["start_time"] = datetime.utcnow() and see if it behaves better?

Nothing changed...

        attrs = self._combine_metadata_with_mode_and_sensor(foreground, background)
        data = self._get_merged_image_data(foreground, background)
        res = super(BackgroundCompositor, self).__call__(data, **kwargs)
        res.attrs.update(attrs)
        res.attrs["sensor"] = "viirs"
        res.attrs["start_time"] = datetime.datetime.utcnow()
        print(res.attrs)
        return res
{'wavelength': None, 'name': 'static_night', '_satpy_id': DataID(name='static_night'), 'prerequisites': [], 'optional_prerequisites': [], 'sensor': 'viirs', 'mode': 'RGB', 'standard_name': 'static_image_spec', 'area': Area ID: None
Description: None
Projection ID: WGS 84
Projection: {'datum': 'WGS84', 'no_defs': 'None', 'proj': 'longlat', 'type': 'crs'}
Number of columns: 4096
Number of rows: 2048
Area extent: (-180.0, -90.0, 180.0, 90.0), 'ancillary_variables': [], 'add_offset': 0.0, 'file_type': 'graphic', 'end_time': datetime.datetime(2023, 12, 21, 2, 42, 55, 850157), 'AREA_OR_POINT': 'Area', 'reader': 'generic_image', 'scale_factor': 1.0, 'start_time': datetime.datetime(2023, 12, 21, 2, 42, 56, 411302), 'nodatavals': (nan, nan, nan)}

So the attrs has been successfully set but KeyError still...Just like I thought before this could be something outside the the whole compositor base.

yukaribbba avatar Dec 21 '23 02:12 yukaribbba

I tried all these just to know if there're any serious problems behind it that could affect other things.

yukaribbba avatar Dec 21 '23 02:12 yukaribbba

Obviously the fact that you said it happens in main suggests something is wrong beyond your changes in this PR. My suggestions for changes in start_time/sensor and adding a standard_name were meant to hopefully reveal (to me at least) where the problem might lie. I might have to try this myself to further debug it, but I'm not sure I'll have time for the next couple weeks. We really need to track down what happens to the DataArray after it is returned by the compositor. Oh...I wonder if the Scene is deleting the composite because it thinks it isn't needed. You could try passing unload=False to the Scene.load call and see if that changes anything.

djhoese avatar Dec 21 '23 03:12 djhoese