Specify botocore config for all boto clients
Summary
Describe your changes.
- Updates all boto3 client instantiations to use the same config object so that they all have the same retry and timeout behavior. This will improve reliability of AWS syncs across the board.
- Moves get_botocore_config() from cartography.intel.aws.ec2.util to cartography.intel.aws.util.common.
- Updates imports of get_botocore_config() to be explicit.
From the documentation, it seems users should be able to use environment variables or ~/.aws/config
https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html#guide-configuration
Broader question:
- Do we really need to override those configs instead of suggesting those values in doc and let the user set those as they prefer?
- Should we use a common function (say
get_botocore_client()incartography/intel/aws/util/boto3.py) to get the client so we can centralize the client creation process in case we need similar client-wide change in future?
Good points. I'll reply in reverse:
Should we use a common function (say get_botocore_client() in cartography/intel/aws/util/boto3.py) to get the client so we can centralize the client creation process in case we need similar client-wide change in future?
If we choose to override botocore, then yes we should centralize it in that one place.
Do we really need to override those configs instead of suggesting those values in doc and let the user set those as they prefer?
This is a hard one. I can see it either way. On the one hand, we know that AWS APIs can be flaky in a way that overriding the boto3 defaults can be helpful with cartography, so for that reason cartography users will benefit from us picking those defaults. But on the other hand, doing this goes against how cartography acts as sort of a 'pass through' for boto3 configuration like how we rely on boto3's search order for finding credentials.
I'm not sure what way to go.
As we are currently overriding some botocore configs, lets continue doing so to avoid any sudden disruption. Lets centralize it so that it is easy to document to tell users how to change the overrides and also in case we decide not to override in distant future.
Closing this PR. I think you're right - the better approach is to provide a link to https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html#guide-configuration in the AWS module configuration section.