HIMAWARI-8/AHI, scene can not load specific composite: "night_ir_with_background"

[sgxl]

Hi, I am going to composite "night_ir_with_background" using HIMAWARI-8/AHI data, but some errors occrus. And I have tried it so many times, so I write a question here for help, thanks.

My code:

    import satpy
    ipath = f'{path_of_file}/*.DAT'  # There's only one moment of data
    file_list = glob.glob(ipath)
    scene = satpy.Scene(filenames=file_list, reader='ahi_hsd')
    scene.load(scene.available_dataset_names())

    # resample the scene before composite
    scene_re = scene.resample(scene.coarsest_area())
    scene_re.load(['night_ir_with_background'])
    scene_re.save_dataset('night_ir_with_background', f'./night_ir_with_background.png')
    # scene_re.load(['true_color_with_night_ir'])

Erroe messages:

The following datasets were not created and may require resampling to be generated: DataID(name='night_ir_with_background')
Traceback (most recent call last):
  File "/home/user01/LIB/miniconda3/lib/python3.9/site-packages/satpy/dataset/data_dict.py", line 169, in __getitem__
    return super(DatasetDict, self).__getitem__(item)
KeyError: 'night_ir_with_background'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "path/t0101a_ahi_rgb_comp.py", line 50, in <module>
    main()
  File "path/t0101a_ahi_rgb_comp.py", line 39, in main
    scene.save_dataset('night_ir_with_background', f'./night_ir_with_background.png')
  File "/home/user01/LIB/miniconda3/lib/python3.9/site-packages/satpy/scene.py", line 1084, in save_dataset
    return writer.save_dataset(self[dataset_id],
  File "/home/user01/LIB/miniconda3/lib/python3.9/site-packages/satpy/scene.py", line 754, in __getitem__
    return self._datasets[key]
  File "/home/user01/LIB/miniconda3/lib/python3.9/site-packages/satpy/dataset/data_dict.py", line 171, in __getitem__
    key = self.get_key(item)
  File "/home/user01/LIB/miniconda3/lib/python3.9/site-packages/satpy/dataset/data_dict.py", line 158, in get_key
    return get_key(match_key, self.keys(), num_results=num_results,
  File "/home/user01/LIB/miniconda3/lib/python3.9/site-packages/satpy/dataset/data_dict.py", line 107, in get_key
    raise KeyError("No dataset matching '{}' found".format(str(key)))
KeyError: "No dataset matching 'DataQuery(name='night_ir_with_background')' found"

Process finished with exit code 1

[sgxl]

To solve this problem, I read the config file: satpy/etc/composites/ahi.yaml, and the related composite is as followed:

  night_ir_with_background:
    compositor: !!python/name:satpy.composites.BackgroundCompositor
    standard_name: night_ir_with_background
    prerequisites:
      - night_ir_alpha
      - _night_background

so I tried the following code to try is any of the prerequisites can not be done?

    scene = satpy.Scene(filenames=file_list, reader='ahi_hsd')
    scene.load(scene.available_dataset_names())
    scene_re = scene.resample(scene.coarsest_area())  # resample the scene for composite.
    scene_re.load(['night_ir_alpha'])
    scene_re.load(['_night_background'])
    scene_re.save_dataset('night_ir_alpha', f'./night_ir_alpha.png')
    scene_re.save_dataset('_night_background', f'./_night_background.png')

All of the prerequisites can be loaded. But the “_night_background” got global data instead the extent of the scene: Composite, night_ir_alpha:

And I tried to load all of the prerequisites before loading composite "night_ir_with_background"

    scene_re = scene.resample(scene.coarsest_area())
    scene_re.load(['night_ir_alpha'])
    scene_re.load(['_night_background'])
    scene_re.save_dataset('night_ir_alpha', f'./night_ir_alpha.png')
    scene_re.save_dataset('_night_background', f'./_night_background.png')
    scene_re.load(['night_ir_with_background'])
    scene_re.save_dataset('night_ir_with_background', f'./night_ir_with_background.png')

Error still occured:

Traceback (most recent call last):
  File "path/t0101a_ahi_rgb_comp.py", line 59, in <module>
    main()
  File "path/t0101a_ahi_rgb_comp.py", line 46, in main
    scene_re.load(['night_ir_with_background'])
  File "/home/user01/LIB/miniconda3/lib/python3.9/site-packages/satpy/scene.py", line 1256, in load
    self._update_dependency_tree(needed_datasets, query)
  File "/home/user01/LIB/miniconda3/lib/python3.9/site-packages/satpy/scene.py", line 1266, in _update_dependency_tree
    comps, mods = load_compositor_configs_for_sensors(self.sensor_names)
  File "/home/user01/LIB/miniconda3/lib/python3.9/site-packages/satpy/scene.py", line 143, in sensor_names
    contained_sensor_names = self._contained_sensor_names()
  File "/home/user01/LIB/miniconda3/lib/python3.9/site-packages/satpy/scene.py", line 158, in _contained_sensor_names
    raise TypeError("Unexpected type in sensor collection")
TypeError: Unexpected type in sensor collection

And I am not sure whether it is a bug, and I have no idea to fix it, so I propose the issue here for help, thanks.

[sgxl]

@jon4than @lamby @mraspaud @adybbroe

[simonrp84]

You need to resa ple the data before you save it, after loading the composite. Coarsest area won't work in this case so you'll need to specify the area of one of the bands. Can't provide an example as am in my phone but if no one else provides one then I'll do so this evening.

[mherbertson]

What @simonrp84 said.

This will resample the night_ir_alpha dataset into the required projection and save in the resolution and extent of the _night_background dataset.

import satpy
from pyresample import create_area_def
ipath = f'{path_of_file}/*.DAT'  # There's only one moment of data
file_list = glob.glob(ipath)
scn = satpy.Scene(filenames=file_list, reader='ahi_hsd')
scn.load(['night_ir_with_background'])

Create area definition:

area_id = 'None'
projection = '+init=EPSG:4326'
resolution = '0.02'
area_def = create_area_def(area_id, projection, resolution=resolution)

The static background uses the depreciated projection = '+init=EPSG:4326' so we have to match it. I'll try and do a PR for that down the track if required.

Resample dataset with area_def:

res_scn = scn.resample(area_def, datasets=['night_ir_alpha'], resampler='nearest')

Resample the scene to either the finest_area or coarsest_area and save:

res_scn2 = res_scn.resample(res_scn.coarsest_area())
res_scn2.save_dataset('night_ir_with_background', f'./night_ir_with_background.png')

Others will chime in if there's a smarter way or there's errors...

[sgxl]

Thanks for your patient answer, @simonrp84 @mherbertson. I have tried the code, and it works. Thanks a lot~

[djhoese]

Thanks for getting this figured out everyone. Before closing this, what do we think about me updating the Scene object to finest_area/coarsest_area ignore "hidden" products that start with a _ when considering areas? This would allow the original code case to work with no extra work. It would not:

1. Use the global area that the final example/solution used.
2. Allow for native resampling which we've seen some users try with this.

[mherbertson]

That would make sense and what users are expecting.

[simonrp84]

@djhoese Only just read through this again, your suggestion to ignore products with a _ at the start sounds good to me.