TooManyRequestsException while syncing AWS APIGateway Rest APIs (retry_mode doesn't help)

[El070422]

Hello everyone,

Cartography 56.0, Centos 7 on AWS EC2, t3.xlarge type, Neo4j 3.5. Cartography was able to get information from several AWS accounts, but one of the accounts doesn't work with Cartography.

Logs: INFO:cartography.intel.aws.apigateway:Syncing AWS APIGateway Rest APIs for region 'us-east-1' in account '459955752294'. ERROR:cartography.sync:Unhandled exception during sync stage 'aws' Traceback (most recent call last): File "/home/centos/.local/lib/python3.8/site-packages/cartography/sync.py", line 74, in run stage_func(neo4j_session, config) File "/home/centos/.local/lib/python3.8/site-packages/cartography/util.py", line 115, in timed return method(*args, **kwargs) File "/home/centos/.local/lib/python3.8/site-packages/cartography/intel/aws/init.py", line 220, in start_aws_ingestion _sync_multiple_accounts( File "/home/centos/.local/lib/python3.8/site-packages/cartography/intel/aws/init.py", line 157, in _sync_multiple_accounts _sync_one_account( File "/home/centos/.local/lib/python3.8/site-packages/cartography/intel/aws/init.py", line 60, in _sync_one_account RESOURCE_FUNCTIONSfunc_name File "/home/centos/.local/lib/python3.8/site-packages/cartography/util.py", line 115, in timed return method(*args, **kwargs) File "/home/centos/.local/lib/python3.8/site-packages/cartography/intel/aws/apigateway.py", line 376, in sync sync_apigateway_rest_apis(neo4j_session, boto3_session, region, current_aws_account_id, update_tag) File "/home/centos/.local/lib/python3.8/site-packages/cartography/util.py", line 115, in timed return method(*args, **kwargs) File "/home/centos/.local/lib/python3.8/site-packages/cartography/intel/aws/apigateway.py", line 366, in sync_apigateway_rest_apis load_rest_api_details(neo4j_session, stages_certificate_resources, current_aws_account_id, aws_update_tag) File "/home/centos/.local/lib/python3.8/site-packages/cartography/util.py", line 115, in timed return method(*args, **kwargs) File "/home/centos/.local/lib/python3.8/site-packages/cartography/intel/aws/apigateway.py", line 296, in load_rest_api_details for api_id, stage, certificate, resource, policy in stages_certificate_resources: File "/home/centos/.local/lib/python3.8/site-packages/cartography/intel/aws/apigateway.py", line 46, in get_rest_api_details resources = get_rest_api_resources(api, client) File "/home/centos/.local/lib/python3.8/site-packages/cartography/util.py", line 115, in timed return method(*args, **kwargs) File "/home/centos/.local/lib/python3.8/site-packages/cartography/intel/aws/apigateway.py", line 93, in get_rest_api_resources for page in response_iterator: File "/home/centos/.local/lib/python3.8/site-packages/botocore/paginate.py", line 253, in iter response = self._make_request(current_kwargs) File "/home/centos/.local/lib/python3.8/site-packages/botocore/paginate.py", line 332, in _make_request return self._method(**current_kwargs) File "/home/centos/.local/lib/python3.8/site-packages/botocore/client.py", line 415, in _api_call return self._make_api_call(operation_name, kwargs) File "/home/centos/.local/lib/python3.8/site-packages/botocore/client.py", line 745, in make_api_call raise error_class(parsed_response, operation_name) botocore.errorfactory.TooManyRequestsException: An error occurred (TooManyRequestsException) when calling the GetResources operation: Too Many Requests

Added the following settings in .aws/config and it didn't help: [profilename] region = *** output = json retry_mode = standard max_attempts = 10

Tried to get AWS APIGateway Rest APIs using AWS Cli and received several items as expected: aws apigateway get-rest-apis --profile profilename --region us-east-1 { "items": [ { "apiKeySource": "HEADER",....

Could you please take a look?

[ramonpetgrave64]

You can try to alter the retry mode in your aws config file. Let us know if that helps. We may want to build this into cartography. https://boto3.amazonaws.com/v1/documentation/api/latest/guide/retries.html

[El070422]

@ramonpetgrave64, I have added retry_mode = standard and max_attempts = 10 to my .aws/config, but it still doesn't work. Maybe there are other ways?

[ramonpetgrave64]

@El070422 I missed that in your original post. Can you try to adaptive mode? Mainly because of it's client-side rate limiting.

[El070422]

Hi @ramonpetgrave64, the same error with retry_mode = adaptive.

[krisek]

max_attempts = 200 retry_mode = adaptive

seem to eliminate the issue in my case

[El070422]

@krisek, no it doesn't work for me, not 200, not 10000, as I wrote above

[achantavy]

Going to close this out since this hasn't been active in a while. Please reopen if you're still seeing the issue. Is it possible that your boto3 session is getting the value from an env var and not the AWS config file? I would look at boto3 docs again and try multiple ways of setting that setting.