IRSA with aws-sdk getting "InvalidToken: The provided token is malformed or otherwise invalid."

[james64]

Describe the bug

This has been already reported (for example #3697) but is closed so I am opening a new one.

Using js aws-sdk with IRSA auth to upload a file to s3 bucket results in InvalidToken: The provided token is malformed or otherwise invalid..

Running aws s3 cp <file> s3://<bucket> in a pod is sucessfull. Running same cmd using js sdk (see reproduction steps) results in the error.

Expected Behavior

Successfull file upload.

Current Behavior

Running reproducing js script (see below) results in this log:

[AWS sts 200 1.94s 0 retries] assumeRoleWithWebIdentity({
  WebIdentityToken: 'eyJ...wJg',
  RoleArn: 'arn:aws:iam::<acccountNum>:role/<roleName>',
  RoleSessionName: 'token-file-web-identity'
})
[AWS s3 400 1.978s 0 retries] putObject({
  Key: 'res/putobject',
  Body: <Buffer 74 65 73 74>,
  Bucket: '<bucket>'
})
(node:9259) UnhandledPromiseRejectionWarning: InvalidToken: The provided token is malformed or otherwise invalid.
    at Request.extractError (/uloha/node_modules/aws-sdk/lib/services/s3.js:711:35)
    at Request.callListeners (/uloha/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
    at Request.emit (/uloha/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
    at Request.emit (/uloha/node_modules/aws-sdk/lib/request.js:686:14)
    at Request.transition (/uloha/node_modules/aws-sdk/lib/request.js:22:10)
    at AcceptorStateMachine.runTo (/uloha/node_modules/aws-sdk/lib/state_machine.js:14:12)
    at /uloha/node_modules/aws-sdk/lib/state_machine.js:26:10
    at Request.<anonymous> (/uloha/node_modules/aws-sdk/lib/request.js:38:9)
    at Request.<anonymous> (/uloha/node_modules/aws-sdk/lib/request.js:688:12)
    at Request.callListeners (/uloha/node_modules/aws-sdk/lib/sequential_executor.js:116:18)
(node:9259) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1)
(node:9259) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

Reproduction Steps

In k8s run pod with IRSA setup. In the pod run the testing js script taken from #3697 using "aws-sdk": "^2.1164.0" as dependency:

const aws = require('aws-sdk');

aws.config.update({
  logger: console,
});

const s3 = new aws.S3({
  region: 'me-south-1',
  params: {
    Bucket: 'oneid-doc-sign-prs'
  },
});

(async function() {
  const response1 = await s3
    .putObject({ Key: 'res/putobject', Body: Buffer.from('test') })
    .promise();
  console.log(response1);
  console.log('done1');

  const response2 = await s3.upload({ Key: 'res/upload', Body: Buffer.from('test') }).promise();
  console.log(response2);
  console.log('done2');
})();

Optionally run same through aws-cli to verify it works.

Possible Solution

No idea to be honest :)

Additional Information/Context

Upload to the bucket is perfectly accessible through all of these methods

• aws console
• js aws sdk when using iam user credentials
• aws cli when using IRSA
• aws cli when using iam user credentials

SDK version used

2.1164.0

Environment details (OS name and version, etc.)

ubuntu image on top of amazon linux os host. K8s 1.21

[james64]

Same issue with 2.1216.0 still.

[ajredniwja]

@james64 thanks for opening this issue and apologies it fell out of queue. I am getting a similar error too, I'll investigate more and post my findings.

[ajredniwja]

Running the script with the latest version doesn't error out for me.

{
  Expiration: 'expiry-date="Sun, 23 Oct 2022 00:00:00 GMT", rule-id="YzZhYjc4MmEtYTAzNS00ZGY0LWIwYmItYWZhisdhknmsid"',
  ETag: '"098f6bcd4621d373cade48789283ef3"',
  VersionId: '_WpryRFHXN09GqqtPjDidajojda93'
}
done1
{
  Expiration: 'expiry-date="Sun, 23 Oct 2022 00:00:00 GMT", rule-id="YzZhYjc4MmEtYTAzNS00ZGY0LWIwYmItYewiouweiojJHidhoj"',
  ETag: '"098f6bcd4621d373cade4e832340940294Kj"',
  VersionId: 'AlhdjSKjdkAL9vpBe2235pE6arQPoEN21',
  Location: 'https://bucket.us-west-2.amazonaws.com/res/upload',
  key: 'res/upload',
  Key: 'res/upload',
  Bucket: 'bucket'
}
done2

Can you please share the steps you follow for setting up the credentials?

[james64]

Thanks for trying this out. Our setup:

1. Spin up EKS cluster
2. Create OIDC IAM identity provider with url from EKS cluster
3. Create policy allowing bucket access
4. Using this terraform module (in this version) to create iam role assumable by OIDC identity. We have a condition to allow only service account from given namespace to assume (these must be equal "oidc.eks.me-south-1.amazonaws.com/id/<clusterid>:sub" = "system:serviceaccount:exampleNs:exampleAccount". Attach bucket policy to this role.
5. In exampleNs create exampleAccount service account and annotate with eks.amazonaws.com/role-arn: <arg_of_oidc_assumable_role>

Then we spin up a pod which just runs ubuntu with long sleep and using example service account. In this pod:

$ apt-get update
$ apt-get install awscli npm vim
$ mkdir test && cd test
$ npm init --yes
$ vim package.json # add dependency for "aws-sdk": "^2.1233.0"
$ vim run.js # copy paste reproduction script verbatim
$ npm install 
$ node run.js
... produces same error as stated above ...
  (node:9134) UnhandledPromiseRejectionWarning: InvalidToken: The provided token is malformed or otherwise invalid.

$ aws --region me-south-1 s3 cp package.json s3://oneid-doc-sign-prs/
upload: ./package.json to s3://oneid-doc-sign-prs/package.json # upload successful

$ env | grep AWS | grep -o '^.*=' # to see that no other AWS envs are set
AWS_DEFAULT_REGION=
AWS_REGION=
AWS_ROLE_ARN=
AWS_WEB_IDENTITY_TOKEN_FILE=

awscli upload immediately afterwards using irsa credentials worked. Node example script with latest version failed. Do you see any difference between our and your setup?

[james64]

@ajredniwja any luck replicating the issue? Maybe you can share description of your setup so I can help spotting the difference.

[BMayhew]

I ran into this issue, I solved it by ensuring that the below environment variables were not set. (we migrated from using the secret key to OIDC) and I had to go unset this in our CI pipeline before succeeding.

AWS_SECRET_KEY AWS_SECRET_ACCESS_KEY

I came to this conclusion based this doc and the order in which things load - https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/auth/DefaultAWSCredentialsProviderChain.html

[james64]

@BMayhew thanks for the post :+1: However we do not have these envs set. List of aws related envs we set is seen in one of my previous post.