Unable to assume ec2 IAM role through local kubernetes

[jpmcb]

Describe the bug

I'm working on kubernetes components that interact with AWS services through ec2 instances that have assumed IAM roles.

I am unable to get aws-sdk-rust calls to resolve correctly due to missing credential errors.

Similar issues: #606 #425

---

Proof of concept rust code:

// [dependencies]
// aws-config = "0.48.0"
// aws-sdk-sts = "0.18.0"
// tokio = { version = "1.20.1", features = ["full"] }

#[tokio::main]
async fn main() {
    let conf = aws_config::from_env().region("us-west-2").load().await;

    let client = aws_sdk_sts::Client::new(&conf);
    let res = client.get_caller_identity().send().await;
    println!("caller_identity = {:#?}", res)
}

When the above program is run on my local hosts, I get the expected results. All seems good here!

$ cargo run
   Compiling sts-poc v0.1.0 (/home/fedora/workspace/ssm-poc/ssm-poc)
    Finished dev [unoptimized + debuginfo] target(s) in 4.91s
     Running `target/debug/sts-poc`
caller_identity = Ok(
    GetCallerIdentityOutput {
        user_id: Some(
            <redacted>,
        ),
        account: Some(
            <redacted>,
        ),
        arn: Some(
            "arn:aws:sts::<redacted>:assumed-role/<redacted>",
        ),
    },
)

Expected Behavior

When my host with the assumed role runs a container, I'd expect aws-sdk-rust to be able to also assume that host's role.

Current Behavior

However, when run through a local kubernetes cluster (kind), I hit the ConstructionFailure(MissingCredentials) error:

Reproduction Steps

Along with the above rust proof of concept program, use the following dockerfile and pod yaml:

---

Dockerfile:

FROM rust
WORKDIR sts-poc
COPY ./ /sts-poc
RUN cargo build --release
CMD ["./target/release/sts-poc"]

Kubernetes pod yaml named run-aws-sdk.yaml

apiVersion: v1
kind: Pod
metadata:
  name: sts-poc
  namespace: default
spec:
  containers:
  - image: sts-poc:demo
    name: sts-proof-of-concept

---

1. Create a kind cluster

$ kind create cluster --name demo

2. Create the container image using docker. Tag it as sts-poc:demo

$ docker build . -t sts-poc:demo

3. Load the image into the kind cluster:

$ kind load docker-image --name demo sts-poc:demo

4. Apply the pod yaml:

$ kubectl apply -f run-aws-poc.yaml

5. Check the logs from the pod. Notice the error.

$ kubectl logs sts-poc
caller_identity = Err(
    ConstructionFailure(
        MissingCredentials,
    ),
)

Possible Solution

Are there things in my environment I'm missing? Other issues on this topic made it seem like this should "just work" ™️

Version

$ cargo tree | rg aws
├── aws-config v0.48.0
│   ├── aws-http v0.48.0
│   │   ├── aws-smithy-http v0.48.0
│   │   │   ├── aws-smithy-types v0.48.0
│   │   ├── aws-smithy-types v0.48.0 (*)
│   │   ├── aws-types v0.48.0
│   │   │   ├── aws-smithy-async v0.48.0
│   │   │   ├── aws-smithy-client v0.48.0
│   │   │   │   ├── aws-smithy-async v0.48.0 (*)
│   │   │   │   ├── aws-smithy-http v0.48.0 (*)
│   │   │   │   ├── aws-smithy-http-tower v0.48.0
│   │   │   │   │   ├── aws-smithy-http v0.48.0 (*)
│   │   │   │   ├── aws-smithy-types v0.48.0 (*)
│   │   │   ├── aws-smithy-http v0.48.0 (*)
│   │   │   ├── aws-smithy-types v0.48.0 (*)
│   ├── aws-sdk-sso v0.18.0
│   │   ├── aws-endpoint v0.48.0
│   │   │   ├── aws-smithy-http v0.48.0 (*)
│   │   │   ├── aws-smithy-types v0.48.0 (*)
│   │   │   ├── aws-types v0.48.0 (*)
│   │   ├── aws-http v0.48.0 (*)
│   │   ├── aws-sig-auth v0.48.0
│   │   │   ├── aws-sigv4 v0.48.0
│   │   │   │   ├── aws-smithy-http v0.48.0 (*)
│   │   │   ├── aws-smithy-http v0.48.0 (*)
│   │   │   ├── aws-types v0.48.0 (*)
│   │   ├── aws-smithy-async v0.48.0 (*)
│   │   ├── aws-smithy-client v0.48.0 (*)
│   │   ├── aws-smithy-http v0.48.0 (*)
│   │   ├── aws-smithy-http-tower v0.48.0 (*)
│   │   ├── aws-smithy-json v0.48.0
│   │   │   └── aws-smithy-types v0.48.0 (*)
│   │   ├── aws-smithy-types v0.48.0 (*)
│   │   ├── aws-types v0.48.0 (*)
│   ├── aws-sdk-sts v0.18.0
│   │   ├── aws-endpoint v0.48.0 (*)
│   │   ├── aws-http v0.48.0 (*)
│   │   ├── aws-sig-auth v0.48.0 (*)
│   │   ├── aws-smithy-async v0.48.0 (*)
│   │   ├── aws-smithy-client v0.48.0 (*)
│   │   ├── aws-smithy-http v0.48.0 (*)
│   │   ├── aws-smithy-http-tower v0.48.0 (*)
│   │   ├── aws-smithy-query v0.48.0
│   │   │   ├── aws-smithy-types v0.48.0 (*)
│   │   ├── aws-smithy-types v0.48.0 (*)
│   │   ├── aws-smithy-xml v0.48.0
│   │   ├── aws-types v0.48.0 (*)
│   ├── aws-smithy-async v0.48.0 (*)
│   ├── aws-smithy-client v0.48.0 (*)
│   ├── aws-smithy-http v0.48.0 (*)
│   ├── aws-smithy-http-tower v0.48.0 (*)
│   ├── aws-smithy-json v0.48.0 (*)
│   ├── aws-smithy-types v0.48.0 (*)
│   ├── aws-types v0.48.0 (*)
├── aws-sdk-sts v0.18.0 (*)

Environment details (OS name and version, etc.)

Fedora 34, kernel 5.11.12, aarch64

Docker build logs

$ docker build . -t sts-poc:demo Sending build context to Docker daemon 1.329GB Step 1/5 : FROM rust ---> ab88ceb21a7d Step 2/5 : WORKDIR sts-poc ---> Using cache ---> ca4a7ec4d0dc Step 3/5 : COPY ./ /sts-poc ---> d6b97d1263e8 Step 4/5 : RUN cargo build --release ---> Running in c8f64db3cb91 Updating crates.io index Downloading crates ... Downloaded base64 v0.13.0 Downloaded libc v0.2.132 Downloaded aws-types v0.48.0 Downloaded socket2 v0.4.7 Downloaded quote v1.0.21 Downloaded rustc_version v0.4.0 Downloaded spin v0.5.2 Downloaded want v0.3.0 Downloaded fnv v1.0.7 Downloaded aws-smithy-query v0.48.0 Downloaded aws-smithy-json v0.48.0 Downloaded aws-smithy-async v0.48.0 Downloaded bytes-utils v0.1.3 Downloaded aws-smithy-http-tower v0.48.0 Downloaded tokio-util v0.7.3 Downloaded futures-core v0.3.24 Downloaded aws-sig-auth v0.48.0 Downloaded itoa v1.0.3 Downloaded pin-utils v0.1.0 Downloaded pin-project-lite v0.2.9 Downloaded http v0.2.8 Downloaded tower-layer v0.3.1 Downloaded aws-sigv4 v0.48.0 Downloaded tower v0.4.13 Downloaded matches v0.1.9 Downloaded aws-smithy-xml v0.48.0 Downloaded aws-smithy-types v0.48.0 Downloaded aws-sdk-sts v0.18.0 Downloaded http-body v0.4.5 Downloaded urlencoding v2.1.2 Downloaded tower-service v0.3.2 Downloaded tokio-stream v0.1.9 Downloaded pin-project v1.0.12 Downloaded tokio-macros v1.8.0 Downloaded ryu v1.0.11 Downloaded httparse v1.8.0 Downloaded tracing v0.1.36 Downloaded hyper v0.14.20 Downloaded futures-util v0.3.24 Downloaded tokio v1.21.0 Downloaded aws-config v0.48.0 Downloaded futures-channel v0.3.24 Downloaded h2 v0.3.14 Downloaded webpki v0.21.4 Downloaded aws-smithy-client v0.48.0 Downloaded zeroize v1.5.7 Downloaded xmlparser v0.13.3 Downloaded ring v0.16.20 Downloaded regex-syntax v0.6.27 Downloaded mio v0.8.4 Downloaded lock_api v0.4.8 Downloaded ct-logs v0.8.0 Downloaded tokio-rustls v0.22.0 Downloaded semver v1.0.13 Downloaded rustls-native-certs v0.5.0 Downloaded hyper-rustls v0.22.1 Downloaded futures-macro v0.3.24 Downloaded either v1.8.0 Downloaded aws-http v0.48.0 Downloaded aws-smithy-http v0.48.0 Downloaded hex v0.4.3 Downloaded regex v1.6.0 Downloaded aho-corasick v0.7.19 Downloaded aws-endpoint v0.48.0 Downloaded fastrand v1.8.0 Downloaded autocfg v1.1.0 Downloaded untrusted v0.7.1 Downloaded unicode-ident v1.0.3 Downloaded try-lock v0.2.3 Downloaded tracing-core v0.1.29 Downloaded tracing-attributes v0.1.22 Downloaded time v0.3.14 Downloaded syn v1.0.99 Downloaded smallvec v1.9.0 Downloaded slab v0.4.7 Downloaded signal-hook-registry v1.4.0 Downloaded sct v0.6.1 Downloaded scopeguard v1.1.0 Downloaded rustls v0.19.1 Downloaded percent-encoding v2.1.0 Downloaded parking_lot_core v0.9.3 Downloaded parking_lot v0.12.1 Downloaded openssl-probe v0.1.5 Downloaded once_cell v1.14.0 Downloaded num_threads v0.1.6 Downloaded num_cpus v1.13.1 Downloaded num-integer v0.1.45 Downloaded memchr v2.5.0 Downloaded log v0.4.17 Downloaded lazy_static v1.4.0 Downloaded indexmap v1.9.1 Downloaded httpdate v1.0.2 Downloaded hashbrown v0.12.3 Downloaded futures-task v0.3.24 Downloaded futures-sink v0.3.24 Downloaded form_urlencoded v1.0.1 Downloaded cfg-if v1.0.0 Downloaded cc v1.0.73 Downloaded bytes v1.2.1 Downloaded proc-macro2 v1.0.43 Downloaded pin-project-internal v1.0.12 Downloaded num-traits v0.2.15 Downloaded aws-sdk-sso v0.18.0 Compiling autocfg v1.1.0 Compiling libc v0.2.132 Compiling proc-macro2 v1.0.43 Compiling unicode-ident v1.0.3 Compiling quote v1.0.21 Compiling syn v1.0.99 Compiling cfg-if v1.0.0 Compiling once_cell v1.14.0 Compiling log v0.4.17 Compiling memchr v2.5.0 Compiling parking_lot_core v0.9.3 Compiling bytes v1.2.1 Compiling pin-project-lite v0.2.9 Compiling scopeguard v1.1.0 Compiling futures-core v0.3.24 Compiling smallvec v1.9.0 Compiling futures-task v0.3.24 Compiling cc v1.0.73 Compiling itoa v1.0.3 Compiling futures-util v0.3.24 Compiling untrusted v0.7.1 Compiling pin-utils v0.1.0 Compiling spin v0.5.2 Compiling fnv v1.0.7 Compiling num_threads v0.1.6 Compiling hashbrown v0.12.3 Compiling futures-channel v0.3.24 Compiling httparse v1.8.0 Compiling futures-sink v0.3.24 Compiling try-lock v0.2.3 Compiling ryu v1.0.11 Compiling tower-service v0.3.2 Compiling httpdate v1.0.2 Compiling base64 v0.13.0 Compiling either v1.8.0 Compiling semver v1.0.13 Compiling percent-encoding v2.1.0 Compiling tower-layer v0.3.1 Compiling openssl-probe v0.1.5 Compiling lazy_static v1.4.0 Compiling fastrand v1.8.0 Compiling regex-syntax v0.6.27 Compiling zeroize v1.5.7 Compiling matches v0.1.9 Compiling hex v0.4.3 Compiling xmlparser v0.13.3 Compiling urlencoding v2.1.2 Compiling tracing-core v0.1.29 Compiling form_urlencoded v1.0.1 Compiling http v0.2.8 Compiling bytes-utils v0.1.3 Compiling lock_api v0.4.8 Compiling tokio v1.21.0 Compiling slab v0.4.7 Compiling num-traits v0.2.15 Compiling indexmap v1.9.1 Compiling num-integer v0.1.45 Compiling aws-smithy-xml v0.48.0 Compiling want v0.3.0 Compiling rustc_version v0.4.0 Compiling aho-corasick v0.7.19 Compiling ring v0.16.20 Compiling aws-types v0.48.0 Compiling socket2 v0.4.7 Compiling mio v0.8.4 Compiling num_cpus v1.13.1 Compiling signal-hook-registry v1.4.0 Compiling time v0.3.14 Compiling http-body v0.4.5 Compiling parking_lot v0.12.1 Compiling regex v1.6.0 Compiling aws-smithy-types v0.48.0 Compiling aws-smithy-query v0.48.0 Compiling aws-smithy-json v0.48.0 Compiling sct v0.6.1 Compiling webpki v0.21.4 Compiling ct-logs v0.8.0 Compiling rustls v0.19.1 Compiling tokio-macros v1.8.0 Compiling tracing-attributes v0.1.22 Compiling futures-macro v0.3.24 Compiling pin-project-internal v1.0.12 Compiling rustls-native-certs v0.5.0 Compiling pin-project v1.0.12 Compiling tracing v0.1.36 Compiling tokio-util v0.7.3 Compiling tokio-stream v0.1.9 Compiling tower v0.4.13 Compiling tokio-rustls v0.22.0 Compiling aws-smithy-async v0.48.0 Compiling h2 v0.3.14 Compiling hyper v0.14.20 Compiling aws-smithy-http v0.48.0 Compiling hyper-rustls v0.22.1 Compiling aws-smithy-http-tower v0.48.0 Compiling aws-sigv4 v0.48.0 Compiling aws-smithy-client v0.48.0 Compiling aws-endpoint v0.48.0 Compiling aws-sig-auth v0.48.0 Compiling aws-http v0.48.0 Compiling aws-sdk-sts v0.18.0 Compiling aws-sdk-sso v0.18.0 Compiling aws-config v0.48.0 Compiling sts-poc v0.1.0 (/sts-poc) Finished release [optimized] target(s) in 1m 20s Removing intermediate container c8f64db3cb91 ---> 7d8956f9af13 Step 5/5 : CMD ["./target/release/sts-poc"] ---> Running in 1d2c777399f4 Removing intermediate container 1d2c777399f4 ---> c5eb809459cf Successfully built c5eb809459cf Successfully tagged sts-poc:demo

[jpmcb]

As another data point, this works as expected when using the aws CLI container:

Container yaml:

apiVersion: v1
kind: Pod
metadata:
  name: aws-cli
spec:
  containers:
  - image: amazon/aws-cli
    command:
      - "aws"
      - "sts"
      - "get-caller-identity"
    name: aws-cli

---

Steps to reproduce

1. Create new kind cluster

$ kind create cluster --name demo

2. Apply the above yaml

$ kubectl apply -f aws-cli.yaml

3. Check logs

$ kubect logs pods/aws-cli
{
    "UserId": "<redacted>",
    "Account": "<redacted>",
    "Arn": "arn:aws:sts::<redacted>:assumed-role/<redacted>"
}

[jmklix]

Can you include detailed logs from when you hit the ConstructionFailure(MissingCredentials) error?

[jpmcb]

Can you include detailed logs from when you hit the ConstructionFailure(MissingCredentials) error?

The only logs that are returned by the pod are:

caller_identity = Err(
    ConstructionFailure(
        MissingCredentials,
    ),
)

I've updated the text of the top level post to be more clear as well as added docker build logs

[mlevkov-ias]

@jpmcb were you able to resolve this at all? I'm having the same issue. From what I understand, the SDK uses IMDSv2 for the EC2 role, not IMDSv1.

[jpmcb]

@mlevkov-ias - I was not able to resolve this: I'm still seeing the same thing.

[jpmcb]

This doesn't appear to be related to IMDSv2 or the hop limit. This is what my instance looks like:

{
    "InstanceId": "i-<redacted>",
    "InstanceMetadataOptions": {
        "State": "pending",
        "HttpTokens": "required",
        "HttpPutResponseHopLimit": 2,
        "HttpEndpoint": "enabled",
        "HttpProtocolIpv6": "disabled",
        "InstanceMetadataTags": "disabled"
    }
}

I get the same results:

❯ k logs pods/sts-poc
caller_identity = Err(
    ConstructionFailure(
        MissingCredentials,
    ),
)