Source Amazon Selling Partner - 403 after 1 hour because security token does not get renewed

[ASchmidtGit]

Environment

Is this your first time deploying Airbyte?: No OS Version / Instance: Ubuntu Memory / Disk: 100 GB Deployment: Docker Airbyte Version: 0.40.26 Source name/version: Amazon Selling Partner v. 0.2.30 Destination name/version: MySQL v. 0.1.20 Step: The issue is happening during sync

Current Behavior

While syncing GET_FLAT_FILE_ALL_ORDERS_DATA_BY_LAST_UPDATE_GENERAL or get_v2_settlement_report_data_flat_file (Report-API) i get an 403 Forbidden error after one hour.

requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://sellingpartnerapi-eu.amazon.com/reports/2021-06-30/documents/amzn1.spdoc.1.4.eu.6dde8110-7f4e-4cdc-8981-9e2f701b54f7.T135ND9ACUT3YX.1117?MarketplaceIds=A1PA6795UKMFR9

Since the duration of the security token (STS) is set to 3600 second (1 hour) by default it needs to be renewed before expiring.

Expected Behavior

Before expiring the security token should be renewed using the refresh token, so reports taking over the default of one hour can be processed.

Logs

Logs for sync of GET_FLAT_FILE_ALL_ORDERS_DATA_BY_LAST_UPDATE_GENERAL

16ce1edb_e6f9_4500_8b47_79e03fa45ebd_logs_102_txt.txt

Logs for sync of get_v2_settlement_report_data_flat_file

16ce1edb_e6f9_4500_8b47_79e03fa45ebd_logs_97_txt.txt

Steps to Reproduce

1. Set up ASP as source and mysql and destination
2. Chose GET_FLAT_FILE_ALL_ORDERS_DATA_BY_LAST_UPDATE_GENERAL or get_v2_settlement_report_data_flat_file
3. enable normalization
4. start syncing
5. wait for an hour

[krisjan-oldekamp]

@ASchmidtGit running into the same issue. Would you be willing to fix this together? Investigated this issue a bit in the past but couldnt find a solution directly then.

[ASchmidtGit]

@krisjan-oldekamp

I already build a workaround which i have to clean up. In my test the sync worked for over 4 hours. I hope i can start cleaning up my fix about next week.

Although im not sure if i instantly start a PR or if i first fix #22555 too, first since this is blocker for us aswell

[krisjan-oldekamp]

@ASchmidtGit Ah great, the fix would be really helpfull on my side as well. Let me know if I can help!

[krisjan-oldekamp]

@ASchmidtGit Any idea when you have time to submit a PR for this? Much appreciated!

[ASchmidtGit]

@krisjan-oldekamp sry for the delay, im currently cleaning up the code and testing

[krisjan-oldekamp]

Hi @ASchmidtGit wondering if you still would need any help with this?

[grok-rs]

@krisjan-oldekamp @ASchmidtGit Hello guys, how do you actually run the sync process I receive an error 403 immediately after 30s with an error requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://sellingpartnerapi-na.amazon.com/reports/2021-06-30/reports. Do I need to add some credentials for an ARN role or something else?

[LincolnWasden]

Same here I get a 403 client error^

[krisjan-oldekamp]

@dromix @LincolnWasden I'm getting the same error on Airbyte Cloud (don't know why), but not in Airbyte OSS

[grok-rs]

@krisjan-oldekamp There are a lot of different streams that may require additional permissions, I'm using several, like orders, orderItems and it works fine, but I didn't find any info about permissions for other streams

[askarpets]

Hello @ASchmidtGit @krisjan-oldekamp, I couldn't reproduce the issue on my side. Could you please confirm it still persists? Thank you!

[askarpets]

Closing the ticket for now as no response from the customer. Please reopen if the issue still persists.