Unable to download from AWS after serialize / deserialize

[varisht-tathya]

Describe the bug Unable to download product from AWS S3 after deserialize.

Code To Reproduce CLI commands or Python code snippet to reproduce the bug.

combined_results = result_cop + result_stac
comb_serial = dag.serialize(combined_results)
recon_serial = dag.deserialize_and_register(comb_serial)
print(recon_serial)
print(recon_serial[-1].properties)
recon_serial[-1].download()

Output Compete output obtained with maximal verbosity.

SearchResult([EOProduct(id=S3A_SL_1_RBT____20240228T031754_20240228T032008_20240228T055340_0134_109_275_0540_PS1_O_NR_004, provider=cop_dataspace),
              EOProduct(id=S3A_SL_1_RBT____20240228T153403_20240228T153703_20240228T173349_0179_109_282_2340_PS1_O_NR_004, provider=cop_dataspace),
              EOProduct(id=S3B_SL_1_RBT____20240228T163606_20240228T163906_20240228T183236_0179_090_140_2340_PS2_O_NR_004, provider=cop_dataspace),
              EOProduct(id=S3B_SL_1_RBT____20240228T163606_20240228T163906_20240229T115700_0179_090_140_2340_PS2_O_NT_004, provider=cop_dataspace),
              EOProduct(id=S3A_SL_1_RBT____20240228T031708_20240228T032008_20240229T131322_0179_109_275_0540_PS1_O_NT_004, provider=cop_dataspace),
              EOProduct(id=S3A_SL_1_RBT____20240228T153403_20240228T153703_20240301T005543_0179_109_282_2340_PS1_O_NT_004, provider=cop_dataspace),
              EOProduct(id=S3B_SL_1_RBT_20240308T034532_20240308T034832_0179_090_261_0540, provider=tathya),
              EOProduct(id=S3B_SL_1_RBT_20240307T162838_20240307T163138_0179_090_254_2340, provider=tathya),
              EOProduct(id=S3A_SL_1_RBT_20240307T031017_20240307T031238_0141_110_004_0540, provider=tathya),
              EOProduct(id=S3A_SL_1_RBT_20240307T030938_20240307T031238_0179_110_004_0540, provider=tathya),
              EOProduct(id=S3A_SL_1_RBT_20240306T155244_20240306T155544_0179_109_382_2340, provider=tathya),
              EOProduct(id=S3A_SL_1_RBT_20240306T033652_20240306T033849_0117_109_375_0540, provider=tathya),
              EOProduct(id=S3A_SL_1_RBT_20240306T033549_20240306T033849_0179_109_375_0540, provider=tathya)])

{'eodag_product_type': 'S3_SLSTR_L1RBT',
 'eodag_provider': 'tathya',
 'eodag_search_intersection': {"coordinates": [[[-87.409999, 36.520188], [-87.401701, 36.520188], [-87.401701, 36.511383], [-87.409999, 36.511383], [-87.409999, 36.520188]]], "type": "Polygon"},
 'abstract': 'SLSTR Level-1 observation mode products consisting of full resolution,\ngeolocated, co-located nadir and along track view, Top of Atmosphere\n(TOA) brightness temperatures (in the case of thermal IR channels) or\nradiances (in the case of visible, NIR and SWIR channels) from all\nSLSTR channels, and quality flags, pixel classification information\nand meteorological annotations\n',
 'instrument': ['SLSTR'],
 'platform': 'Sentinel-3',
 'platformSerialIdentifier': 'Sentinel-3A',
 'processingLevel': 'L1',
 'keywords': 'SLSTR,SENTINEL,SENTINEL3,S3,S3A,S3B,L1,L1RBT,RBT',
 'sensorType': 'ATMOSPHERIC',
 'license': 'proprietary',
 'title': 'S3A_SL_1_RBT_20240306T033549_20240306T033849_0179_109_375_0540',
 'missionStartDate': '2016-02-16T00:00:00Z',
 'productType': 'Sentinel-3-Collection',
 'orbitNumber': 375,
 'orbitDirection': 'ascending',
 'cloudCover': 56.146701,
 'startTimeFromAscendingNode': '2024-03-06T03:37:19.143250Z',
 'completionTimeFromAscendingNode': '2024-03-06T03:38:49.143250Z',
 'downloadLink': 'https://stg.tathya.earth/stac/collections/Sentinel-3-Collection/items/S3A_SL_1_RBT_20240306T033549_20240306T033849_0179_109_375_0540',
 'storageStatus': 'ONLINE',
 's3:gsd': {'S1-S6': 500, 'S7-S9 and F1-F2': 1000},
 's3:land': 99.991319,
 'safe_id': 'S3A_SL_1_RBT____20240306T033549_20240306T033849_20240307T132223_0179_109_375_0540_PS1_O_NT_004.SEN3',
 's3:coastal': 0.000417,
 's3:cosmetic': 26.761458,
 's3:saturated': 0.0,
 's3:duplicated': 4.78691,
 'start_datetime': '2024-03-06T03:35:49.143250Z',
 's3:out_of_range': 2.6e-05,
 's3:product_name': 'slstr-rbt',
 's3:product_type': 'SL_1_RBT___',
 's3:saline_water': 0.008681,
 's3:tidal_region': 0.030243,
 'sat:absolute_orbit': 41928,
 's3:fresh_inland_water': 3.489306,
 's3:processing_timeliness': 'NT',
 'sat:platform_international_designator': '2016-011A',
 'id': 'S3A_SL_1_RBT_20240306T033549_20240306T033849_0179_109_375_0540'}

2024-03-13 16:40:48,648 eodag.download.base              [INFO    ] (tid=129759731722048) Download url: https://stg.tathya.earth/stac/collections/Sentinel-3-Collection/items/S3A_SL_1_RBT_20240306T033549_20240306T033849_0179_109_375_0540
2024-03-13 16:40:49,874 eodag.download.aws               [WARNING ] (tid=129759731722048) Unexpected error: An error occurred (PermanentRedirect) when calling the ListObjects operation: The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.
2024-03-13 16:40:49,875 eodag.download.aws               [WARNING ] (tid=129759731722048) Skipping stg/stac/collections/Sentinel-3-Collection/items/S3A_SL_1_RBT_20240306T033549_20240306T033849_0179_109_375_0540

---------------------------------------------------------------------------
AuthenticationError                       Traceback (most recent call last)
Cell In[68], line 1
----> 1 recon_serial[-1].download()

File ~/anaconda3/envs/arosicsupgrade/lib/python3.9/site-packages/eodag/api/product/_product.py:335, in EOProduct.download(self, progress_callback, wait, timeout, **kwargs)
    329     self.location = urllib.parse.unquote(self.location)
    331 progress_callback, close_progress_callback = self._init_progress_bar(
    332     progress_callback
    333 )
--> 335 fs_path = self.downloader.download(
    336     self,
    337     auth=auth,
    338     progress_callback=progress_callback,
    339     wait=wait,
    340     timeout=timeout,
    341     **kwargs,
    342 )
    344 # close progress bar if needed
    345 if close_progress_callback:

File ~/anaconda3/envs/arosicsupgrade/lib/python3.9/site-packages/eodag/plugins/download/aws.py:361, in AwsDownload.download(self, product, auth, progress_callback, **kwargs)
    359 # could not auth on any bucket
    360 if not authenticated_objects:
--> 361     raise AuthenticationError(", ".join(auth_error_messages))
    363 # downloadable files
    364 product_chunks = []

AuthenticationError: An error occurred (PermanentRedirect) when calling the ListObjects operation: The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.

Environment:

• Python version: Python 3.8.8
• EODAG version: 2.11.0

Additional context Add any other context about the bug here. The provider tathya has the products stored in AWS S3 bucket. I am able to download it directly when the product is not serialized / deserialized.

[sbrunato]

Hello @varisht-tathya , sorry but we were not able to reproduce this issue with eodag providers using AwsDownload. Can you share the configuration of your provider tathya and tell us how to get credentials ?

[varisht-tathya]

It is a privately hosted STAC server. The product is saved as a STAC item, with the individual bands saved to AWS S3 bucket.