satpy
satpy copied to clipboard
pytroll
Memory overflow during sun-zenith correction for MSIL1C
Describe the bug When processing Sentinel 2 images into true color RGB composite, the sun-zenith correction causes the process to be killed due to the lack of memory. The machine has ample memory and the problem seems to occur only for older images (~2020 with no problems for this years images) . Also, the false color composite is processed correctly.
Can anyone provide some guidance how to narrow down the problem please?
To Reproduce
description = "ATKAstd in SouthPolarSt"
# Area extent as a tuple (lower_left_x, lower_left_y, upper_right_x, upper_right_y)
area_extent =(85000, 2135000, 180000, 2205000)
proj_id = 'EPSG:3031'
area_extent = (-315239.5, 2093800.2999999998, -265239.5, 2143800.3)
areadefs['ATKAstd'] = AreaDefinition("ATKAstd", "SouthPolarSt", description, proj_id, 5000, 5000, area_extent)
p = f"data_sentinel2/{args.date}"
files = find_files_and_readers(
base_dir=p,
reader='msi_safe')
print(files)
scene = Scene(filenames=files)
plot_path = Path('plots/')
plot_path.mkdir(parents=True, exist_ok=True)
scene.load(['true_color'])
new_scn = scene.resample(areadefs["ATKAstd"])
outname = str(plot_path / f"{scene.start_time.strftime('%Y%m%d_%H%M%S')}_sentinel2_true_color")
new_scn.save_dataset('true_color', writer='geotiff',
filename=outname+'tif')
Expected behavior The sun-zenith correction is successful and the true color composite is processed.
Actual results Text output of actual results or error messages including full tracebacks if applicable.
[DEBUG: 2023-12-03 14:31:34 : rasterio._base] Nodata success: 0, Nodata value: -10000000000.000000
[DEBUG: 2023-12-03 14:31:34 : rasterio._base] Dataset <open DatasetReader name='data_sentinel2/20201129/S2B_MSIL1C_20201129T091009_N0500_R021_T29DNB_20230323T232840.SAFE/GRANULE/L1C_T29DNB_A019495_20201129T091006/IMG_DATA/T29DNB_20201129T091009_B03.jp2' mode='r'> is started.
[DEBUG: 2023-12-03 14:31:34 : rasterio.env] Exiting env context: <rasterio.env.Env object at 0xffff6b4a9030>
[DEBUG: 2023-12-03 14:31:34 : rasterio.env] Cleared existing <rasterio._env.GDALEnv object at 0xffff6b4a8e80> options
[DEBUG: 2023-12-03 14:31:34 : rasterio._env] Stopped GDALEnv <rasterio._env.GDALEnv object at 0xffff6b4a8e80>.
[DEBUG: 2023-12-03 14:31:34 : rasterio.env] Exiting outermost env
[DEBUG: 2023-12-03 14:31:34 : rasterio.env] Exited env context: <rasterio.env.Env object at 0xffff6b4a9030>
/home/ubuntu/s2-atka/s2-env/lib/python3.10/site-packages/satpy/readers/msi_safe.py:319: RuntimeWarning: Mean of empty slice
angles = np.nanmean(np.dstack(arrays), -1)
/home/ubuntu/s2-atka/s2-env/lib/python3.10/site-packages/satpy/readers/msi_safe.py:319: RuntimeWarning: Mean of empty slice
angles = np.nanmean(np.dstack(arrays), -1)
[DEBUG: 2023-12-03 14:33:21 : satpy.modifiers.geometry] Applying sun zen correction
[DEBUG: 2023-12-03 14:33:21 : satpy.modifiers.geometry] Apply the effective solar atmospheric path length correction method by Li and Shibata
[DEBUG: 2023-12-03 14:33:25 : satpy.modifiers.geometry] Applying sun zen correction
[DEBUG: 2023-12-03 14:33:25 : satpy.modifiers.geometry] Apply the standard sun-zenith correction [1/cos(sunz)]
[DEBUG: 2023-12-03 14:33:26 : h5py._conv] Creating converter from 7 to 5
[DEBUG: 2023-12-03 14:33:26 : h5py._conv] Creating converter from 5 to 7
[DEBUG: 2023-12-03 14:33:26 : h5py._conv] Creating converter from 7 to 5
[DEBUG: 2023-12-03 14:33:26 : h5py._conv] Creating converter from 5 to 7
[INFO: 2023-12-03 14:36:20 : satpy.modifiers.atmosphere] Removing Rayleigh scattering with atmosphere 'us-standard' and aerosol type 'rayleigh_only' for 'B04'
[INFO: 2023-12-03 14:36:20 : pyspectral.rayleigh] Atmosphere chosen: us-standard
[WARNING: 2023-12-03 14:36:20 : pyspectral.rayleigh] Inconsistent sensor/satellite input - sensor set to msi
[DEBUG: 2023-12-03 14:36:20 : pyspectral.rayleigh] LUT filename: /home/ubuntu/.local/share/pyspectral/rayleigh_only/rayleigh_lut_us-standard.h5
[DEBUG: 2023-12-03 14:36:20 : pyspectral.rsr_reader] Filename: /home/ubuntu/.local/share/pyspectral/rsr_msi_Sentinel-2B.h5
[DEBUG: 2023-12-03 14:36:20 : pyspectral.rsr_reader] Filename: /home/ubuntu/.local/share/pyspectral/rsr_msi_Sentinel-2B.h5
[DEBUG: 2023-12-03 14:36:20 : h5py._conv] Creating converter from 3 to 5
[DEBUG: 2023-12-03 14:36:20 : pyspectral.rsr_reader] No detectors found - assume only one...
[DEBUG: 2023-12-03 14:36:20 : pyspectral.rsr_reader] No detectors found - assume only one...
[DEBUG: 2023-12-03 14:36:20 : pyspectral.rsr_reader] No detectors found - assume only one...
[DEBUG: 2023-12-03 14:36:20 : pyspectral.rsr_reader] No detectors found - assume only one...
[DEBUG: 2023-12-03 14:36:20 : pyspectral.rsr_reader] No detectors found - assume only one...
[DEBUG: 2023-12-03 14:36:20 : pyspectral.rsr_reader] No detectors found - assume only one...
[DEBUG: 2023-12-03 14:36:20 : pyspectral.rsr_reader] No detectors found - assume only one...
[DEBUG: 2023-12-03 14:36:20 : pyspectral.rsr_reader] No detectors found - assume only one...
[DEBUG: 2023-12-03 14:36:20 : pyspectral.rsr_reader] No detectors found - assume only one...
[DEBUG: 2023-12-03 14:36:20 : pyspectral.rsr_reader] No detectors found - assume only one...
[DEBUG: 2023-12-03 14:36:20 : pyspectral.rsr_reader] No detectors found - assume only one...
[DEBUG: 2023-12-03 14:36:20 : pyspectral.rsr_reader] No detectors found - assume only one...
[DEBUG: 2023-12-03 14:36:20 : pyspectral.rsr_reader] No detectors found - assume only one...
[DEBUG: 2023-12-03 14:36:20 : pyspectral.rayleigh] Band name: B04 Effective wavelength: 0.664457um
[DEBUG: 2023-12-03 14:37:16 : satpy.modifiers.geometry] Applying sun zen correction
[DEBUG: 2023-12-03 14:37:16 : satpy.modifiers.geometry] Apply the effective solar atmospheric path length correction method by Li and Shibata
[INFO: 2023-12-03 14:37:35 : satpy.modifiers.atmosphere] Removing Rayleigh scattering with atmosphere 'us-standard' and aerosol type 'rayleigh_only' for 'B03'
[INFO: 2023-12-03 14:37:35 : pyspectral.rayleigh] Atmosphere chosen: us-standard
[WARNING: 2023-12-03 14:37:35 : pyspectral.rayleigh] Inconsistent sensor/satellite input - sensor set to msi
[DEBUG: 2023-12-03 14:37:35 : pyspectral.rayleigh] LUT filename: /home/ubuntu/.local/share/pyspectral/rayleigh_only/rayleigh_lut_us-standard.h5
[DEBUG: 2023-12-03 14:37:35 : pyspectral.rsr_reader] Filename: /home/ubuntu/.local/share/pyspectral/rsr_msi_Sentinel-2B.h5
[DEBUG: 2023-12-03 14:37:35 : pyspectral.rsr_reader] Filename: /home/ubuntu/.local/share/pyspectral/rsr_msi_Sentinel-2B.h5
[DEBUG: 2023-12-03 14:37:35 : pyspectral.rsr_reader] No detectors found - assume only one...
[DEBUG: 2023-12-03 14:37:35 : pyspectral.rsr_reader] No detectors found - assume only one...
[DEBUG: 2023-12-03 14:37:35 : pyspectral.rsr_reader] No detectors found - assume only one...
[DEBUG: 2023-12-03 14:37:35 : pyspectral.rsr_reader] No detectors found - assume only one...
[DEBUG: 2023-12-03 14:37:35 : pyspectral.rsr_reader] No detectors found - assume only one...
[DEBUG: 2023-12-03 14:37:35 : pyspectral.rsr_reader] No detectors found - assume only one...
[DEBUG: 2023-12-03 14:37:35 : pyspectral.rsr_reader] No detectors found - assume only one...
[DEBUG: 2023-12-03 14:37:35 : pyspectral.rsr_reader] No detectors found - assume only one...
[DEBUG: 2023-12-03 14:37:35 : pyspectral.rsr_reader] No detectors found - assume only one...
[DEBUG: 2023-12-03 14:37:35 : pyspectral.rsr_reader] No detectors found - assume only one...
[DEBUG: 2023-12-03 14:37:35 : pyspectral.rsr_reader] No detectors found - assume only one...
[DEBUG: 2023-12-03 14:37:35 : pyspectral.rsr_reader] No detectors found - assume only one...
[DEBUG: 2023-12-03 14:37:35 : pyspectral.rsr_reader] No detectors found - assume only one...
[DEBUG: 2023-12-03 14:37:35 : pyspectral.rayleigh] Band name: B03 Effective wavelength: 0.558175um
[DEBUG: 2023-12-03 14:38:31 : satpy.modifiers.geometry] Applying sun zen correction
[DEBUG: 2023-12-03 14:38:31 : satpy.modifiers.geometry] Apply the effective solar atmospheric path length correction method by Li and Shibata
Killed
Environment Info:
- OS: Ubuntu 22.04 on AWS r6g.2xlarge (64GB RAM)
- Satpy Version: 0.45.0
- PyResample Version: 1.27.1
- Readers and writers dependencies (when relevant): msi_safe: ok
