Running with IAM Profile on EKS

[IliaGe]

Hey, I'm running the nginx-s3-gateway over EKS, while the POD is not able to work when I'm attaching the SA to an IAM Role. I'm getting the following error: 2022/05/30 14:11:38 [error] 59#59: *396 js exception: Error: unhandled promise rejection: Error: read timed out, client: 127.0.0.1, server: , request: "GET /3_5.txt HTTP/1.1", subrequest: "/aws/credentials/retrieve", host: "127.0.0.1:7000"

We use the same framework as explained: https://docs.aws.amazon.com/eks/latest/userguide/specify-service-account-role.html The POD is aware of the SA, but can't get it to work :( AWS_ROLE_ARN='arn:aws:iam::xxx:role/nginx-s3-gateway-D2A48-nginx-s3-gateway' AWS_SIGS_VERSION='4' AWS_WEB_IDENTITY_TOKEN_FILE='/var/run/secrets/eks.amazonaws.com/serviceaccount/token'

[dekobon]

@IliaGe thank you for the bug report. I do not quite understand the context of the error above. This is my understanding:

1. You are trying to setup the gateway to run on EKS.
2. You want the gateway to authenticate using an instance credential.

Questions:

1. What does the acronym SA mean?
2. What are the environment variables AWS_ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE intended purpose?

I believe that you are the first one to try to get the gateway working on EKS using some form of instance credentials. It would be great to get your assistance to better understand what we need to do to get it working.

[IliaGe]

Hey @dekobon, SA == Service Account I'm using a Service-Account config on EKS, that is attached to an IAM Role. https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html

The SDK should work in this setup, without any specific config. but this doesn't work on the nginx

[IliaGe]

https://aws.amazon.com/blogs/opensource/introducing-fine-grained-iam-roles-service-accounts/

[dekobon]

The nginx S3 gateway does not use the AWS SDK. It is just an example configuration of nginx and some njs (NGINX Javascript) put together that works to proxy requests backed by S3, so support for different AWS service account authentication methods is added one at a time. If you would like to jump in and offer a PR to add support such that EKS works, it would be much appreciated.

[mvanholsteijn]

Hi @dekobon , I discovered that the nginx-s3-gateway does not work correctly with a IAM role bound to a kubenetes service account.

The credentials have to be retrieved by a call to sts assumerolewithwebcredentials using the AWS_ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE environment variables as shown below. The call does not have to be signed.

curl -X POST https://sts.amazonaws.com/ \
-d Version=2011-06-15 \
-d Action=AssumeRoleWithWebIdentity \
-d RoleArn=$AWS_ROLE_ARN \
-d WebIdentityToken=$(<$AWS_WEB_IDENTITY_TOKEN_FILE) \
-d RoleSessionName=$(hostname)

Inspiration for a suggested fix can be found https://github.com/mvanholsteijn/nginx-s3-gateway' I did not have time to test it properly.

[maliankurh]

Thanks @mvanholsteijn for the suggested fix. I tested the code and made few changes to make it work. I am not a developer so there might be a better way to code it. That being said the code is working fine for me without any issues.

If anyone wants to refer it here is the updated code https://github.com/maliankurh/nginx-s3-gateway

[dekobon]

Thank you for sharing your fix @maliankurh. Would you be able to submit it as a pull request?

[maliankurh]

Sure I will create a pull request and let you know

[maliankurh]

Hello @dekobon , I have created pull request https://github.com/nginxinc/nginx-s3-gateway/pull/52.

[tooptoop4]

i'm hitting 2 different issues with this:

1. https://github.com/nginxinc/nginx-s3-gateway/blob/master/common/docker-entrypoint.d/00-check-for-required-env.sh#L33 complains about not passing in access/secret key

2022/11/07 03:17:34 [notice] 90#90: start worker process 91
2022/11/07 03:17:34 [notice] 90#90: start cache manager process 92
2022/11/07 03:17:34 [notice] 90#90: start cache loader process 93
2022/11/07 03:17:47 [info] 91#91: *1 js: Cached credentials are expired or not present, requesting new ones
2022/11/07 03:18:35 [notice] 93#93: http file cache: /var/cache/nginx/s3_proxy 0.000M, bsize: 4096
2022/11/07 03:18:35 [notice] 90#90: signal 17 (SIGCHLD) received from 93
2022/11/07 03:18:35 [notice] 90#90: cache loader process 93 exited with code 0
2022/11/07 03:18:35 [notice] 90#90: signal 29 (SIGIO) received
2022/11/07 03:18:47 [info] 91#91: *1 js: Could not assume role using web identity: {}
2022/11/07 03:18:47 [error] 91#91: *1 auth request unexpected status: 500 while SSL handshaking to fetch target, client: 127.0.0.1, server: , request: "GET / HTTP/1.1", host: "localhost"
127.0.0.1 - - [07/Nov/2022:03:18:47 +0000] "GET / HTTP/1.1" 500 170 "-" "curl/7.74.0" "-"

[jnodorp-jaconi]

For future reference, as this issue comes up when searching for IAM:

1. https://github.com/nginxinc/nginx-s3-gateway/blob/master/common/docker-entrypoint.d/00-check-for-required-env.sh#L33 complains about not passing in access/secret key

I addressed that with #70 / #71. As a workaround: I overwrite the file like this in our Kubernetes deployment:

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-s3-gw-fix
data:
  00-check-for-required-env.sh: |
    # Put the fixed version from #71 here!
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-s3-gateway
spec:
  template:
    spec:
      containers:
        - name: nginx-s3-gateway
          [...]
          volumeMounts:
            - name: fix-check
              mountPath: /docker-entrypoint.d/00-check-for-required-env.sh
              subPath: "00-check-for-required-env.sh"
      volumes:
        - name: fix-check
          configMap:
            name: nginx-s3-gw-fix-check
            items:
              - key: "00-check-for-required-env.sh"
                path: "00-check-for-required-env.sh"
                mode: 0700

[...]
2022/11/07 03:18:47 [error] 91#91: *1 auth request unexpected status: 500 while SSL handshaking to fetch target, client: 127.0.0.1, server: , request: "GET / HTTP/1.1", host: "localhost"

This can be fixed by setting the environment variableJS_TRUSTED_CERT_PATH to /etc/ssl/certs/Amazon_Root_CA_1.pem.