sen2like icon indicating copy to clipboard operation
sen2like copied to clipboard

Published 20 hours ago • verified publisher icon senbox-org

Skip product LC09_L1TP_... due to sen2cor failure

Open ValentinaHutter opened this issue 6 months ago • 3 comments
trafficstars

I noticed an issue with some of the Landsat files in the sen2cor processing, which looks like:

2025-05-15 03:49:11.568 | 2025-05-15 01:49:11 \| P-23 \| T-140419704599168 \| INFO     \| sen2like             - Process of L8/L9 products finish |  
-- | -- | --
  | 2025-05-15 03:49:11.568 | 2025-05-15 01:49:11 \| P-23 \| T-140419704599168 \| INFO     \| sen2like             - End process for input product <LOCAL PATH>/SEN2LIKE/Landsat9/190/27/LC09_L1TP_190027_20240928_20240928_02_T1 |  
  | 2025-05-15 03:49:11.568 | 2025-05-15 01:49:11 \| P-23 \| T-140419704599168 \| INFO     \| sen2like             - Skip product LC09_L1TP_190027_20240928_20240928_02_T1 due to sen2cor failure |  
  | 2025-05-15 03:49:11.568 | core.sen2cor_client.sen2cor_client.Sen2corError: Sen2Cor error: Cannot get output product from <LOCAL PATH>/SEN2LIKE/production/1747233708/sen2cor/LC09_L1TP_190027_20240928_20240928_02_T1 |  
  | 2025-05-15 03:49:11.568 | raise Sen2corError(f"Sen2Cor error: Cannot get output product from {sen2cor_output_dir}") |  
  | 2025-05-15 03:49:11.568 | File "/usr/local/sen2like/core/sen2cor_client/sen2cor_client.py", line 121, in run |  
  | 2025-05-15 03:49:11.568 | ^^^^^^^^^^^^^^^^^^^^^^^^ |  
  | 2025-05-15 03:49:11.568 | sen2cor.run(s2l_product), |  
  | 2025-05-15 03:49:11.568 | File "/usr/local/sen2like/./sen2like.py", line 119, in pre_process_atmcor |  
  | 2025-05-15 03:49:11.568 | Traceback (most recent call last): |  
  | 2025-05-15 03:49:11.567 | 2025-05-15 01:49:11 \| P-23 \| T-140419704599168 \| WARNING  \| sen2like             - sen2cor raises an error |  
  | 2025-05-15 03:49:11.566 | 2025-05-15 01:49:11 \| P-23 \| T-140419704599168 \| ERROR    \| sen2cor_client       - Sen2Cor error: Cannot get output product |  
  | 2025-05-15 03:49:10.673 |   |  
  | 2025-05-15 03:49:10.672 | 2025-05-15T01:49:10.672  sen2like-zrchg-process-3323463923 049590 [ INFO] Progress[%]: 100.00 : Application terminated successfully. |  
  | 2025-05-15 03:48:55.561 | 2025-05-15T01:48:55.560  sen2like-zrchg-process-3323463923 049590 [ INFO] Progress[%]: 81.93 : exporting band WATER_VAPOUR, elapsed time[s]: 0.936, total: 0:03:38.675069

While some of the Landsat files are processed successfully: LC08_L1GT_190027_20240819_20250514_02_T2, LC09_L1TP_190026_20240811_20240811_02_T1, ... These ones did not complete: LC09_L1TP_190027_20240928_20240928_02_T1, LC08_L1TP_190027_20240920_20240927_02_T1, LC08_L1GT_190027_20240819_20240823_02_T2, ...

do you have any idea why this could be the case?

ValentinaHutter avatar May 15 '25 07:05 ValentinaHutter

Dear Valentina,

Thank you for your message.

From the extracts of the log you sent, the line:

3323463923 049590 [ INFO] Progress[%]: 100.00 : Application terminated successfully. |

Seems to indicate a successful Sen2cor processing. Maybe a file access too fast that raise an error? It looks more like a file access issue to me.

Do you have the full log of Sen2like and of Sen2cor processing ?

Best regards, Jérôme

De : ValentinaHutter @.> Envoyé : jeudi 15 mai 2025 09:03 À : senbox-org/sen2like @.> Cc : Subscribed @.***> Objet : [senbox-org/sen2like] Skip product LC09_L1TP_... due to sen2cor failure (Issue #20)

TELESPAZIO Security Awareness - This is an incoming mail from an EXTERNAL SENDER. Please verify sender before you open attachments or access links.

[https://avatars.githubusercontent.com/u/85164505?s=20&v=4]ValentinaHutter created an issue (senbox-org/sen2like#20)https://github.com/senbox-org/sen2like/issues/20

I noticed an issue with some of the Landsat files in the sen2cor processing, which looks like:

2025-05-15 03:49:11.568 | 2025-05-15 01:49:11 | P-23 | T-140419704599168 | INFO | sen2like - Process of L8/L9 products finish |

-- | -- | --

| 2025-05-15 03:49:11.568 | 2025-05-15 01:49:11 | P-23 | T-140419704599168 | INFO | sen2like - End process for input product <LOCAL PATH>/SEN2LIKE/Landsat9/190/27/LC09_L1TP_190027_20240928_20240928_02_T1 |

| 2025-05-15 03:49:11.568 | 2025-05-15 01:49:11 | P-23 | T-140419704599168 | INFO | sen2like - Skip product LC09_L1TP_190027_20240928_20240928_02_T1 due to sen2cor failure |

| 2025-05-15 03:49:11.568 | core.sen2cor_client.sen2cor_client.Sen2corError: Sen2Cor error: Cannot get output product from <LOCAL PATH>/SEN2LIKE/production/1747233708/sen2cor/LC09_L1TP_190027_20240928_20240928_02_T1 |

| 2025-05-15 03:49:11.568 | raise Sen2corError(f"Sen2Cor error: Cannot get output product from {sen2cor_output_dir}") |

| 2025-05-15 03:49:11.568 | File "/usr/local/sen2like/core/sen2cor_client/sen2cor_client.py", line 121, in run |

| 2025-05-15 03:49:11.568 | ^^^^^^^^^^^^^^^^^^^^^^^^ |

| 2025-05-15 03:49:11.568 | sen2cor.run(s2l_product), |

| 2025-05-15 03:49:11.568 | File "/usr/local/sen2like/./sen2like.py", line 119, in pre_process_atmcor |

| 2025-05-15 03:49:11.568 | Traceback (most recent call last): |

| 2025-05-15 03:49:11.567 | 2025-05-15 01:49:11 | P-23 | T-140419704599168 | WARNING | sen2like - sen2cor raises an error |

| 2025-05-15 03:49:11.566 | 2025-05-15 01:49:11 | P-23 | T-140419704599168 | ERROR | sen2cor_client - Sen2Cor error: Cannot get output product |

| 2025-05-15 03:49:10.673 | |

| 2025-05-15 03:49:10.672 | 2025-05-15T01:49:10.672 sen2like-zrchg-process-3323463923 049590 [ INFO] Progress[%]: 100.00 : Application terminated successfully. |

| 2025-05-15 03:48:55.561 | 2025-05-15T01:48:55.560 sen2like-zrchg-process-3323463923 049590 [ INFO] Progress[%]: 81.93 : exporting band WATER_VAPOUR, elapsed time[s]: 0.936, total: 0:03:38.675069

While some of the Landsat files are processed successfully: LC08_L1GT_190027_20240819_20250514_02_T2, LC09_L1TP_190026_20240811_20240811_02_T1, ... These ones did not complete: LC09_L1TP_190027_20240928_20240928_02_T1, LC08_L1TP_190027_20240920_20240927_02_T1, LC08_L1GT_190027_20240819_20240823_02_T2, ...

do you have any idea why this could be the case?

— Reply to this email directly, view it on GitHubhttps://github.com/senbox-org/sen2like/issues/20, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ARV5ZQPE3X3F4D5ADZMO5JT26Q3ZFAVCNFSM6AAAAAB5FH45EKVHI2DSMVQWIX3LMV43ASLTON2WKOZTGA3DKMJYGMZTKOA. You are receiving this because you are subscribed to this thread.Message ID: @.@.>>

-WARNING: This message contains confidential and/or proprietary information which may be subject to privilege or immunity and which is intended for the use of its addressee only. Should you receive this message in error, you are kindly requested to inform the sender and to definitively remove it from any paper or electronic format. Any other use of this e-mail is strictly forbidden. Thank you in advance for your cooperation.

Please consider the environment before printing this e-mail.

telespazio-tim avatar May 15 '25 08:05 telespazio-tim

Thanks for the quick response!

Unfortunately, this seems to be the full log, for some reason, it does not show any other information about the traceback...

ValentinaHutter avatar May 15 '25 08:05 ValentinaHutter

Hi Valentina,

Could you send the full sen2like log so that we could search for the sen2like command that triggers your sen2cor process (through the sen2cor_client) ?

  • Information on the platform (linux, windows)
  • Configuration file used (e.g. options used, etc. etc.)
  • Any information that may be useful (e.g. in case of retry the same LC_09 product, does it fails again ?)

For sen2cor log and configuration files, they should be available in a folder that looks like this:

ls /data/wd/31UFS/1746435773/sen2cor/LC09_L1TP_197025_20230404_20230404_02_T1/LC09_L2TP_197025_20230404_20250505_02_T1

  • L2A_CAL_AC_GIPP.xml
  • L2A_CAL_SC_GIPP.xml
  • L2A_QUALITY.xml
  • LC09_L2TP_197025_20230404_20250505_02_T1_report.xml
  • sen2cor_gipp_31UFS.xml

Best regards, Jérôme

De : ValentinaHutter @.> Envoyé : jeudi 15 mai 2025 10:24 À : senbox-org/sen2like @.> Cc : sen2like @.>; Comment @.> Objet : Re: [senbox-org/sen2like] Skip product LC09_L1TP_... due to sen2cor failure (Issue #20)

TELESPAZIO Security Awareness - This is an incoming mail from an EXTERNAL SENDER. Please verify sender before you open attachments or access links.

[Image supprimée par l'expéditeur.]ValentinaHutter left a comment (senbox-org/sen2like#20)https://github.com/senbox-org/sen2like/issues/20#issuecomment-2882993390

Thanks for the quick response!

Unfortunately, this seems to be the full log, for some reason, it does not show any other information about the traceback...

— Reply to this email directly, view it on GitHubhttps://github.com/senbox-org/sen2like/issues/20#issuecomment-2882993390, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ARV5ZQMPCQ2AFENLUAA6RUD26RFJBAVCNFSM6AAAAAB5FH45EKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDQOBSHE4TGMZZGA. You are receiving this because you commented.Message ID: @.***>

-WARNING: This message contains confidential and/or proprietary information which may be subject to privilege or immunity and which is intended for the use of its addressee only. Should you receive this message in error, you are kindly requested to inform the sender and to definitively remove it from any paper or electronic format. Any other use of this e-mail is strictly forbidden. Thank you in advance for your cooperation.

Please consider the environment before printing this e-mail.

telespazio-tim avatar May 20 '25 13:05 telespazio-tim