Can't' obtain AWS credentials inside k8s pod in China region

[GenrikhFetischev]

Checkboxes for prior research

• [X] I've gone through Developer Guide and API reference
• [X] I've checked AWS Forums and StackOverflow.
• [X] I've searched for previous similar issues and didn't find any solution.

Describe the bug

Hello! Could you please help me with the following problem: I need to obtain AWS creds from token file inside a pod in k8s cluster. Previously I used aws-sdk v2 for that purpose and it worked, the problem occured after migration to sdk v3. The error is:

InvalidIdentityTokenException: No OpenIDConnect provider found in your account for https://oidc.eks.cn-north-1.amazonaws.com.cn/id/<ID>
    at de_InvalidIdentityTokenExceptionRes (/path/node_modules/@aws-sdk/client-sts/dist-cjs/protocols/Aws_query.js:436:23)
    at de_AssumeRoleWithWebIdentityCommandError (/path/node_modules/@aws-sdk/client-sts/dist-cjs/protocols/Aws_query.js:219:25)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async /path/node_modules/@smithy/middleware-serde/dist-cjs/deserializerMiddleware.js:7:24
    at async /path/node_modules/@smithy/middleware-retry/dist-cjs/retryMiddleware.js:27:46
    at async /path/node_modules/@aws-sdk/middleware-logger/dist-cjs/loggerMiddleware.js:7:26
    at async /path/node_modules/@aws-sdk/client-sts/dist-cjs/defaultStsRoleAssumers.js:58:33
    at async file:///x/index.js:9:17 {
  '$fault': 'client',
  '$metadata': {
    httpStatusCode: 400,
    requestId: '<ID>',
    extendedRequestId: undefined,
    cfId: undefined,
    attempts: 1,
    totalRetryDelay: 0
  },
  Error: {
    Type: 'Sender',
    Code: 'InvalidIdentityToken',
    Message: 'No OpenIDConnect provider found in your account for https://oidc.eks.cn-north-1.amazonaws.com.cn/id/<ID>',
    message: 'No OpenIDConnect provider found in your account for https://oidc.eks.cn-north-1.amazonaws.com.cn/id/<ID>'
  },
  RequestId: '<ID>',
  xmlns: 'https://sts.amazonaws.com/doc/2011-06-15/'
}

I went through the following troubleshoot process - https://repost.aws/knowledge-center/eks-troubleshoot-oidc-and-irsa and looks like everything is ok according to the checklist.

The problem is observed only in China region clusters, in the rest regions everything is ok.

SDK version number

@aws-sdk/[email protected]

Which JavaScript Runtime is this issue in?

Node.js

Details of the browser/Node.js/ReactNative version

v20.5.1

Reproduction Steps

I'm not sure I can provide a k8s cluster to reproduce the bug. The code which trying to retrieve creds is the following:

const inPodCreds = fromTokenFile()();

Observed Behavior

I'm getting the following error:

InvalidIdentityTokenException: No OpenIDConnect provider found in your account for https://oidc.eks.cn-north-1.amazonaws.com.cn/id/<ID>
    at de_InvalidIdentityTokenExceptionRes (/path/node_modules/@aws-sdk/client-sts/dist-cjs/protocols/Aws_query.js:436:23)
    at de_AssumeRoleWithWebIdentityCommandError (/path/node_modules/@aws-sdk/client-sts/dist-cjs/protocols/Aws_query.js:219:25)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async /path/node_modules/@smithy/middleware-serde/dist-cjs/deserializerMiddleware.js:7:24
    at async /path/node_modules/@smithy/middleware-retry/dist-cjs/retryMiddleware.js:27:46
    at async /path/node_modules/@aws-sdk/middleware-logger/dist-cjs/loggerMiddleware.js:7:26
    at async /path/node_modules/@aws-sdk/client-sts/dist-cjs/defaultStsRoleAssumers.js:58:33
    at async file:///x/index.js:9:17 {
  '$fault': 'client',
  '$metadata': {
    httpStatusCode: 400,
    requestId: '<ID>',
    extendedRequestId: undefined,
    cfId: undefined,
    attempts: 1,
    totalRetryDelay: 0
  },
  Error: {
    Type: 'Sender',
    Code: 'InvalidIdentityToken',
    Message: 'No OpenIDConnect provider found in your account for https://oidc.eks.cn-north-1.amazonaws.com.cn/id/<ID>',
    message: 'No OpenIDConnect provider found in your account for https://oidc.eks.cn-north-1.amazonaws.com.cn/id/<ID>'
  },
  RequestId: '<ID>',
  xmlns: 'https://sts.amazonaws.com/doc/2011-06-15/'
}

Expected Behavior

I expect to get AWS credentials from provider.

Possible Solution

No response

Additional Information/Context

The problem is observed only in China region clusters, in the rest regions everything is ok. The same code can retrieve apps in others cluster. Code with aws-sdk v2 works in China region k8s clusters as well, the problem only being observed with aws-sdk-3

[yenfryherrerafeliz]

Hi @GenrikhFetischev, sorry to hear about your issues. There are different reasons why you could be getting this error. But, lets try to figure out the basics first:

• Can you please go to AWS Console -> IAM -> Identity providers and do you see the provider we get in the error there?, which is oidc.eks.cn-north-1.amazonaws.com.cn/id/<ID>. If you do not see it there is because it was deactivated.
• Can you please make sure OIDC is enabled by running the following: aws eks describe-cluster --name "your-cluster-name" --query "cluster.identity.oidc.issuer".

Please let me know the results.

Thanks!

[github-actions[bot]]

This issue has not received a response in 1 week. If you still think there is a problem, please leave a comment to avoid the issue from automatically closing.

[GenrikhFetischev]

Hello @yenfryherrerafeliz! Thanks for your response. As I mentioned I went through the following troubleshoot process - https://repost.aws/knowledge-center/eks-troubleshoot-oidc-and-irsa that includes steps you advised to start with.

In particular:

• Check if you have an existing IAM OIDC provider for your cluster
• Check if your IAM role has an attached needful IAM policy with required permissions
• Verify that the IAM role trust relations are correctly set
• Check if you created a service account
• Verify that the service account has the correct IAM role annotations
• Verify that you correctly specified the serviceAccountName in your pod
• Check the environment variables and permissions
• Verify that the application uses a supported AWS SDK
• Check the pod user and group
• Recreate pods
• Verify that the audience is correct
• Verify that you configured the correct thumbprint
• For the AWS China Region, check the AWS_DEFAULT_REGION environment variable

And everything looks as it intended to be. Also, I mentioned that authorization works in aws sdk v2 in the exactly same cluster, OIDC providers exists, active and work.

[aBurmeseDev]

Hi @GenrikhFetischev - sorry for the long silence here. Is this still an ongoing issue for you?

The error suggests that the OpenID Connect (OIDC) provider is not set up correctly in your AWS account for the China (Beijing) region (cn-north-1).

The OIDC provider is a crucial component for Kubernetes clusters using IAM Roles for Service Accounts (IRSA) to work with EKS. It allows the Kubernetes service accounts to assume AWS IAM roles and obtain temporary AWS credentials.

Since you mentioned that the problem only occurs in the China region clusters and not in other regions, it could be related to a specific configuration or setup required for the China regions.

I'd be happy to further assist you if you're still working on this project. Best, John

[GenrikhFetischev]

Hello! @aBurmeseDev Is it possible that with OIDC provider which is not correctly set up AWS SDK v2 works as expected?

[aBurmeseDev]

To better understand and address the issue, we would need to examine your specific configuration and setup, particularly the regions you have configured. I tried to reproduce the problem in the cn-north-1 region but was unable to do so successfully.

[GenrikhFetischev]

Ok, let me try to reproduce that one more time. It's been a while since I had that problem, may be in that year the problem was fixed. I don't know current state because we have been using SDK v2 due to the problem

[github-actions[bot]]

This issue has not received a response in 1 week. If you still think there is a problem, please leave a comment to avoid the issue from automatically closing.

[github-actions[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs and link to relevant comments in this thread.