Make AWS work with env variables, not just files

[danielsaporo]

It's quite convenient for various reasons to run Cartography inside a Docker container. When doing so, the obvious question becomes how to pass in the AWS region and credentials.

One way to do this is to mount the .aws folder containing config and credentials into the container. This works.

Another option could be to pass in the necessary configuration/credentials as environment variables. This includes:

• AWS_DEFAULT_REGION
• AWS_ACCESS_KEY_ID
• AWS_SECRET_ACCESS_KEY

Unfortunately, this doesn't work because of Cartography's reliance on AWS profiles, which seem to be native to the configuration/credentials file.

It would be nice if Cartography could be made to work by passing the necessary configuration/credentials as environment variables, even if this might restrict it to a single account.

[achantavy]

Not sure if I completely understand, but cartography uses boto3 for AWS calls and boto3 uses both env vars and files with this order of precedence: https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#configuring-credentials

If you are able to pass env vars to the Dockerfile, then cartography will create the default boto3 session here: https://github.com/lyft/cartography/blob/739187096a012a56888b598da6799c24c561e669/cartography/intel/aws/init.py#L201 which will then detect creds based on the order in the link above.

Please let me know if I missed something but I don't think we do anything to specifically require files.

[danielsaporo]

If you are able to pass env vars to the Dockerfile

I've tried that, and it doesn't work, because there are several places that create the boto3 Session with a profile, e.g.:

https://github.com/lyft/cartography/blob/739187096a012a56888b598da6799c24c561e669/cartography/intel/aws/init.py#L156

That's what I meant by "Cartography's reliance on AWS profiles" earlier.

[achantavy]

Ah, now I understand.

Unfortunately I can't commit to working on this, but I am supportive of accepting and merging in a PR that enables this scenario as long as the existing profile-reliant flow remains intact.

Side note: As additional reference, here are our setup instructions where they do say the config file profiles are necessary: https://lyft.github.io/cartography/modules/aws/config.html#modules-aws-config--page-root. The PR would have to update this as well.