artillery
artillery copied to clipboard
artilleryio
Receiving InvalidClientTokenId Error when running Artillery lambda test
I am trying to load test a small application that I have running in an AWS Gov Cloud EC2 instance. I have followed all the documentation from the Artillery website on how to configure the proper IAM roll for the Lambda function that is created by the Artillery SDK. As soon as I run the artillery command in the directory that contains the aws credentials for my AWS environment I receive a specific error:
λ Creating AWS Lambda function...
⠦ /usr/local/share/.config/yarn/global/node_modules/aws-sdk/lib/protocol/query.js:50
resp.error = util.error(new Error(), {
^
InvalidClientTokenId: The security token included in the request is invalid.
at Request.extractError (/usr/local/share/.config/yarn/global/node_modules/aws-sdk/lib/protocol/query.js:50:29)
at Request.callListeners (/usr/local/share/.config/yarn/global/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
at Request.emit (/usr/local/share/.config/yarn/global/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
at Request.emit (/usr/local/share/.config/yarn/global/node_modules/aws-sdk/lib/request.js:686:14)
at Request.transition (/usr/local/share/.config/yarn/global/node_modules/aws-sdk/lib/request.js:22:10)
at AcceptorStateMachine.runTo (/usr/local/share/.config/yarn/global/node_modules/aws-sdk/lib/state_machine.js:14:12)
at /usr/local/share/.config/yarn/global/node_modules/aws-sdk/lib/state_machine.js:26:10
at Request.<anonymous> (/usr/local/share/.config/yarn/global/node_modules/aws-sdk/lib/request.js:38:9)
at Request.<anonymous> (/usr/local/share/.config/yarn/global/node_modules/aws-sdk/lib/request.js:688:12)
at Request.callListeners (/usr/local/share/.config/yarn/global/node_modules/aws-sdk/lib/sequential_executor.js:116:18) {
code: 'InvalidClientTokenId',
time: 2023-04-04T15:56:22.908Z,
requestId: 'a737e4bf-3f96-4669-842c-77744126baa7',
statusCode: 403,
retryable: false,
retryDelay: 59.471072044108396
}
--
Version info:
<versionVERSION INFO:
Artillery: 2.0.0-31
Node.js: v16.19.1
OS: darwin
>
Running this command:
<artillery run 'our file path' --platform aws:lambda --platform-opt region=us-gov-east-1>
I expected to see this happen:
explanation
Instead, this happened:
A successful artillery test...?
Files being used:
<relevant yaml/js/csv go here>
hi @pracinejr 👋 this could be due to a misconfiguration of AWS credentials on your machine.
Do you have the official AWS CLI installed? (https://aws.amazon.com/cli/) Could you try running the following command to check that it's configured correctly:
aws sts get-caller-identity
Hey! thank you for the quick response!
So when I run the command you provided I get a json object output with my UserId, Account, and Arn that have been configured for my machine to access our AWS CLI. Is that what I should expect? Just a note in case it was easily missed in the initial description, I am working in the AWS GovCloud. Could that be causing any issues? Also, as a follow up I am using the AWS IAM role designated by the artillery documentation listed below with the necessary adjustments to fit our environment:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "CreateOrGetLambdaRole",
"Effect": "Allow",
"Action": [
"iam:CreateRole",
"iam:GetRole"
],
"Resource": "arn:aws:iam::123456789000:role/artilleryio-default-lambda-role"
},
{
"Sid": "CreateLambdaPolicy",
"Effect": "Allow",
"Action": [
"iam:CreatePolicy",
"iam:AttachRolePolicy"
],
"Resource": "arn:aws:iam::123456789000:policy/artilleryio-lambda-policy"
},
{
"Sid": "SQSPermissions",
"Effect": "Allow",
"Action": [
"sqs:*"
],
"Resource": "arn:aws:sqs:*:123456789000:artilleryio*"
},
{
// ListQueues does cannot be scoped to individual resources
// https://docs.aws.amazon.com/service-authorization/latest/reference/list_amazonsqs.html#amazonsqs-queue
"Sid": "SQSListQueues",
"Effect": "Allow",
"Action": [
"sqs:ListQueues"
],
"Resource": "*"
},
{
"Sid": "LambdaPermissions",
"Effect": "Allow",
"Action": [
"lambda:InvokeAsync",
"lambda:CreateFunction",
"lambda:DeleteFunction"
],
"Resource": "arn:aws:lambda:*:123456789000:function:artilleryio-*"
},
{
"Sid": "S3Permissions",
"Effect": "Allow",
"Action": [
"s3:DeleteObject",
"s3:GetObject",
"s3:PutObject",
"s3:ListBucket",
"s3:GetLifecycleConfiguration",
"s3:PutLifecycleConfiguration"
],
"Resource": [
"arn:aws:s3:::artilleryio-test-data-*",
"arn:aws:s3:::artilleryio-test-data-*/*"
]
}
]
}
Great, if the sts command works, it means the AWS profile is set up correctly!
I don't think running in AWS GovCloud would cause issues (none I'm aware of anyway). Looking at our Lambda guide, it does not seem like we actually document that Artillery expects the AWS_PROFILE environment variable to be set in order to pass the credentials information to the underlying AWS SDK. Could you check that it's set and points to the AWS profile you want to use?
Unfortunately, I'm not sure where or how to check that. Would you be able to provide me with a set of steps to do so?
No worries! If you're not setting an AWS profile explicitly, then you're probably using the default one, which is either called default or govcloud if you followed this guide: https://docs.aws.amazon.com/govcloud-us/latest/UserGuide/configure-using-cli.html
Could you try running the following:
export AWS_PROFILE=default
artillery run # the rest of the command
and if that doesn't work, could you try:
export AWS_PROFILE=govcloud
artillery run # the rest of the command
Ok so I found the name of the AWS profile we are using, which is set explicitly. I then ran the command you asked: export AWS_PROFILE="the name of our user" and then ran the artillery command and I received a different error:
λ Creating AWS Lambda function...
⠙ node:internal/process/promises:279
triggerUncaughtException(err, true /* fromPromise */);
^
Error: connect EHOSTDOWN 'address' - Local ('address')
at internalConnect (node:net:1041:16)
at defaultTriggerAsyncIdScope (node:internal/async_hooks:464:18)
at node:net:1134:9
at processTicksAndRejections (node:internal/process/task_queues:78:11) {
message: 'Missing credentials in config, if using AWS_CONFIG_FILE, set AWS_SDK_LOAD_CONFIG=1',
errno: -64,
code: 'CredentialsError',
syscall: 'connect',
address: 'address',
port: 'port',
time: 2023-04-06T15:07:16.123Z,
originalError: {
message: 'Could not load credentials from any providers',
errno: -64,
code: 'CredentialsError',
syscall: 'connect',
address: 'address',
port: 'port',
time: 2023-04-06T15:07:16.123Z,
originalError: {
message: 'EC2 Metadata roleName request returned error',
errno: -64,
code: 'EHOSTDOWN',
syscall: 'connect',
address: 'address',
port: 'port',
time: 2023-04-06T15:07:16.123Z,
originalError: {
errno: -64,
code: 'EHOSTDOWN',
syscall: 'connect',
address: 'address',
port: 'port',
message: 'connect EHOSTDOWN 'address' - Local ('address')'
}
}
}
}
Hey Peter, here's to hoping you see this message!
(PS don't tell my boss I'm posting here on GitHub lol 😁)
After reading what Hassy had to say, "Artillery expects the AWS_PROFILE environment variable to be set in order to pass the credentials information to the underlying AWS SDK."
It got me thinking that we never tested this during our session. If you're up to it, can you do the following test please?
In your .~/aws/credentials file, make sure that you specify both a [default] profile, as well as a named profile, e.g [petertest] with your programmatic keys (Also, remember to check if they're still valid by calling aws sts get-caller-identity if you're using an ASIA access key - these are temporary!).
Please make sure both the default profile and named profile uses the same credentials (for redundancy, if Artillery falls back to default profile we'll at least know which keys are being used to sign the API calls).
Then, set your AWS_PROFILE environment variable to the same name as your named profile per Hassy's guidance;
export AWS_PROFILE=petertest
Then (very important), note whether you access keys are configured in the Credentials or Config file. I believe I saw that your credentials were set in the config file during our session, but I am not 100% sure.
-> If it's in the credential file, then you're good. -> If it's in the config file, then you need to set an additional environment variable - AWS_SDK_LOAD_CONFIG=1 (this is noted in the error responses from Artillery, it doesn't specify whether it's an env variable so we'll assume so)
export AWS_SDK_LOAD_CONFIG=1
Then remember to add the "--platform-opt region=us-gov-east-1" region parameter to your command when executing it.
Also, please check in your config file if you have a profile set up for [default] and [petertest], as ideally you should rather configure the programmatic keys in the Credentials file and the additional profile parameters such as the region and the output in the Config file. If you do, then make sure that it's region is also set to us-gov-east-1.
Example credentials file:
[default] aws_access_key_id = aws_secret_access_key = aws_session_token =
[petertest] aws_access_key_id = aws_secret_access_key = aws_session_token =
Example config file:
[default] region=us-gov-east-1 output=json
[petertest] region=us-gov-east-1 output=json
If, under these hopefully more refined circumstances, you still see the same InvalidClientTokenId error code then I can only think of two other possible reasons;
- The X.509 certificate is invalid (which we did not get to during our session, but in all fairness if the X.509 cert used by your environment was invalid then we would see other API calls fail as well, not just Artillery) or
- AWS access key ID provided does not exist in our records (which stands to reason that the region value in the actual API request payload being sent to the AWS service endpoints is a non gov-cloud region and in that case it makes sense since gov-cloud and non gov-cloud programmatic credentials are not inter-usable)
I hope this makes sense and is able to share some knowledge @hassy & @pracinejr =)
