S3 access via IAM instance profile

[DmitriyStoyanov]

For the moment there is no ability to use IAM instance profile for access to s3 bucket as it is not described in documentation Could you add this ability, please? It will increase security and secret key and access key will be not be used in this integration, only IAM Instance profile attached to EC2 instance where sitespeed.io is running.

[soulgalore]

Cool, this would be a good first issue for someone to take on. We use https://www.npmjs.com/package/aws-sdk and then the documentation needs to be updated: https://www.sitespeed.io/documentation/sitespeed.io/s3/

[DmitriyStoyanov]

hm, but in code I see next part https://github.com/sitespeedio/sitespeed.io/blob/f8fb300ce78cb58e4a0c400c17e2cb2e029b5d57/lib/plugins/s3/index.js#L97

    if (this.s3Options.key || this.s3Options.secret) {
      throwIfMissing(this.s3Options, ['key', 'secret'], 's3');
    }

in that case it will not throw exception if it is not set. and from documentation here https://docs.aws.amazon.com/sdk-for-javascript/v2/developer-guide/loading-node-credentials-iam.html and here https://docs.aws.amazon.com/sdk-for-javascript/v2/developer-guide/setting-credentials-node.html It should use IAM instance profile, am I right?

[soulgalore]

You mean it will be automatically picked up? Yep could be, haven't looked into the docs. Right now as you say it needs to have the cli params but that could be fixed.

[soulgalore]

@DmitriyStoyanov are you able to test if it work if you just remove the throw if missing? If you can verify that it works, I can do the tuning to fix it.

[DmitriyStoyanov]

I'm busy right now, and will be available for that next week, then i guess i can do that.

[hutemai]

Hi @soulgalore, I've tested it. Actually it works using IAM instance role without specifying key id and access key.

[DmitriyStoyanov]

yes, it is tested and looks fine. So possibly need to add such info into documentation

[barryperez]

Hi there! Just wonder how you guys were able to get this working? I've been struggling with this one today.

1. I've validated that I can write to the S3 bucket using AWS CLI from my host machine - so the correct IAM role is applied to the EC2 instance.

2. I run the sitespeed.io Docker image in interactive mode, and having added an AWS config file as follows:

[default]
role_arn = arn:aws:iam::xxxx:role/role-name
credential_source = Ec2InstanceMetadata
region = yyy

3. I have then set the following environment variables

# set | grep AWS
AWS_CONFIG_FILE=/tmp/.aws/config
AWS_SDK_LOAD_CONFIG=1

4. I run SiteSpeed via start.sh, as follows:

./start.sh --s3.endpoint="http://s3.xxx.amazonaws.com/" --s3.region="region-name-here" --s3.bucketname="sitespeed-results-xxxxx" -n 1 https://www.sitespeed.io

The test runs and I then get the following error on upload:

[2021-09-09 14:01:10] INFO: Uploading /sitespeed-result/www.sitespeed.io/2021-09-09-14-00-50 to S3 bucket sitespeed-results-xxxxx, this can take a while ... [2021-09-09 14:01:15] ERROR: Could not upload to S3 CredentialsError: Missing credentials in config, if using AWS_CONFIG_FILE, set AWS_SDK_LOAD_CONFIG=1 at IncomingMessage. (/usr/src/app/node_modules/aws-sdk/lib/util.js:904:34) at IncomingMessage.emit (events.js:387:35) at IncomingMessage.emit (domain.js:470:12) at endReadableNT (internal/streams/readable.js:1317:12) at processTicksAndRejections (internal/process/task_queues.js:82:21)

Maybe I'm missing something simple but it looks like that the SDK isn't picking up the config? Thanks in advance!

[barryperez]

Addendum - works fine if I use host network to run the container - it's only when using the bridge adapter that it fails. (i.e. standard bridge one or one that was created to simulate different network conditions)

Will try to look further when I have time - but maybe this rings a bell for someone in the meanwhile?

[soulgalore]

I can have try later this week, I want to get this documented.