amazon-eks-pod-identity-webhook icon indicating copy to clipboard operation
amazon-eks-pod-identity-webhook copied to clipboard

Documentation should call out use of WebIdentityTokenCredentialProvider in the default CrentialProvider chain and non-root user USER directive

Open rbranche opened this issue 3 years ago • 0 comments

What would you like to be added:

Documentation should:

  1. Call out use of WebIdentityTokenCredentialProvider in the default CrentialProvider chain.
  2. Dockerfile must explicitly specify the non-root USER directive

This request is specifically for the Java SDK, but I'm sure other SDKs would hit the same issues

Why is this needed:

At Adobe, we are converting all of our Java services to start using this instead of kube2iam, so I went through the process of converting one of our services.

I managed to get the token loaded in the container with the OIDC provider and ServiceAccount setup, but struggled for a while with the service actually being able to use it with the SDK's Default CredentialProvider chain (it was falling all the way back to Step 6 instead of stopping at Step 3 like expected): https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/credentials.html#credentials-chain

I finally found the documentation for SDK V2 which mentioned that STS was required to be on the classpath: https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/auth/credentials/WebIdentityTokenFileCredentialsProvider.html

Use of this credentials provider requires the 'sts' module to be on the classpath.

So, I updated the service to include that dependency, but it still wasn't working. The CredentialProvider was being used, but it was failing to access the token file due to permission issues. The service does run as a non-root user, but in K8s 1.19 and above this shouldn't be a problem with no securityContext.fsGroup specification.

I found that the service's image file did not have a USER directive specified which made Kubernetes deny access. Once I added the USER directive (for the non-root user) to the Dockerfile, everything started working.

tldr

  1. The Amazon AWS Java SDK STS module must be an explicit dependency for any application that uses the AWS Java SDK and IAM roles for Kubernetes ServiceAccounts

V2

        <dependency>
            <groupId>software.amazon.awssdk</groupId>
            <artifactId>sts</artifactId>
        </dependency>
        <dependency>
            <groupId>software.amazon.awssdk</groupId>
            <artifactId>auth</artifactId>
        </dependency>

V1

        <dependency>
            <groupId>com.amazonaws</groupId>
            <artifactId>aws-java-sdk-sts</artifactId>
        </dependency>
  1. If the service is running as a non-root user and there is no securityContext.fsGroup, then there must be an explicit USER directive in the Dockerfile for Kubernetes 1.19+ to allow access to the WebIdentity token file.

rbranche avatar Oct 04 '21 22:10 rbranche