support for AWS Organizations / organisational unit

[tbugfinder]

Is your feature request related to a problem? Please describe. AWS accounts can be managed using AWS Organizations and grouped with organisational units (OU). This account structure also supports a login to a "central" AWS account and assuming roles within member accounts. It would be very helpful to configure a single login AWS account to login into and the assuming a role while connecting to all AWS accounts within the organization or just a subset (OU).

Describe the solution you'd like Configure login credentials to the "billing" account and a role being used when connecting to member accounts.

Describe alternatives you've considered Basically the profile can be also prepared using scripts and query member account IDs of an OU.

Additional context N/A

[cbruno10]

Hey @tbugfinder , thanks for the request!

We currently don't have any integrations with AWS Organizations for connections, but as you mentioned in your considered alternatives, there actually is a project, https://github.com/happy240/steampipe-conn-generator-for-aws-organization, that is helpful in creating Steampipe AWS configuration files based on your organization + OUs. If you give it a try, please let us know how it goes!

In terms of Steampipe + AWS Orgs integration, I had a few questions:

• Are there any projects out there that do AWS Orgs integration well that you use today or know of?
• Do you use multiple organizations today, or mainly one?
• How many accounts are you looking to connect to?
• Are you looking to connect to specific OUs/accounts, or all OUs/accounts in the organization? If there are specific ones, are there any patterns in how they're organized, e.g., names, tags?
• If you could setup an organization, would all member accounts have the IAM role that Orgs uses to access them? From what I remember, the role is created by default, but users can delete this role once the account is created. Or is there another way that an Organizations management account can access member accounts (even if the IAM role doesn't exist)?

[tbugfinder]

Hi, thanks for pointing my to that project which works well or can be adjusted to my needs.

• Are there any projects out there that do AWS Orgs integration well that you use today or know of?

Well, I'd say even quite some AWS native services do not integrate well with AWS Orgs. In that sense I'm not aware of 3rd tools that integrate with AWS Orgs (of the top of my head).

• Do you use multiple organizations today, or mainly one?

Currently it's a single org.

• How many accounts are you looking to connect to?

roughly 500

• Are you looking to connect to specific OUs/accounts, or all OUs/accounts in the organization? If there are specific ones, are there any patterns in how they're organized, e.g., names, tags?

My current need is a specific OU incl. the subtree of it. I like the idea of selecting OUs. based on tags which are in place.

• If you could setup an organization, would all member accounts have the IAM role that Orgs uses to access them? From what I remember, the role is created by default, but users can delete this role once the account is created. Or is there another way that an Organizations management account can access member accounts (even if the IAM role doesn't exist)?

IMHO, a well architected Org has configured a delegated admin account incl. a cross-account IAM role which should/could have RO permissions - I wouldn't use the default role but force deployment of a dedicated role (deny changes&removal using service control policies)

[tbugfinder]

Few days ago, I'd created a config utilizing above repo and it worked.

Today I noticed that I can update steampipe and the plugin, however having that in place postgres dies when that config for given number of AWS accounts is in place.

# exec steampipe
$ steampipe plugin list
Error: Plugin Listing failed - unexpected EOF

# /var/log/messages
kernel: [1421009.143840] Out of memory: Kill process 122078 (postgres) score 228 or sacrifice child
kernel: [1421009.143924] Killed process 122078 (postgres), UID xxxx, total-vm:4159772kB, anon-rss:2750540kB, file-rss:452kB, shmem-rss:25424kB

# tail .steampipe/logs/da*


2022-05-13 14:17:59.001 UTC [119280] LOG:  server process (PID 8330) was terminated by signal 9: Killed
2022-05-13 14:17:59.001 UTC [119280] LOG:  terminating any other active server processes
2022-05-13 14:17:59.121 UTC [11813] LOG:  connection received: host=127.0.0.1 port=45726
2022-05-13 14:17:59.126 UTC [11813] LOG:  PID 8330 in cancel request did not match any process
2022-05-13 14:17:59.159 UTC [119280] LOG:  all server processes terminated; reinitializing
2022-05-13 14:17:59.232 UTC [11815] LOG:  connection received: host=127.0.0.1 port=45728

[cbruno10]

@tbugfinder Does that command (steampipe plugin list) consistently fail? Do other commands work, like steampipe query?

Also, if you reduce the number of accounts by 25%, 50%, etc., do the commands start to work again? We have some users who have connections for several hundred AWS accounts today, so we believe this can work, but there may be some local configuration settings that need to be adjusted.

@kaidaguerre Do you have any other suggestions around making a large number of connections work, e.g., limiting the number of connections at a time, adjusting caching time, etc.?

[tbugfinder]

Actually it failed multiple times. I can increase OS memory and do some other tests.

[cbruno10]

@tbugfinder While running trying other tests/scenarios, can you please try reducing the number of actions, and then trying to increase that number each time? We've had successful reports of 300 account connections or so, and we're actively trying to improve the CLI + plugin to handle large number of connections better, so any data points are helpful in this area.

Hopefully, it should help you run queries as well for some or most of your accounts in the meantime.

Thanks!

[github-actions[bot]]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 30 days.

[tbugfinder]

it's still on me to replay....

[cbruno10]

Related to closed issue, https://github.com/turbot/steampipe-plugin-aws/issues/1115, which shares similar requirements.

[github-actions[bot]]

'This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 30 days.'

[github-actions[bot]]

'This issue was closed because it has been stalled for 90 days with no activity.'