[AWS] support AWS Pod Identity authentication - AWS SDK bump

[balonik]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Use Cases

To be able to use AWS Pod Identity authentication method for vector pods running in AWS.

ref: https://docs.aws.amazon.com/eks/latest/userguide/pod-id-minimum-sdk.html https://github.com/awslabs/aws-sdk-rust/releases/tag/release-2024-03-13

Attempted Solutions

N/A - current AWS SDK version used in Vector does not support Pod Identity.

Proposal

Bump AWS SDK versions - aws_config at least 1.1.8 which adds Pod Identity credentials provider.

References

No response

Version

0.38.0

[mzupan]

just a heads up that this is really needed as pod identiy is nicer to handle gitops. If anyone runs across this error while trying to use pod identity the cause is needing to upgrade the SDK

vector-cm84k vector 2024-08-17T17:17:50.242676Z  WARN sink{component_kind="sink" component_id=kubernetes_s3 component_type=aws_s3}:request{request_id=1}:lazy_load_identity: aws_config::meta::credentials::chain: provider failed to provide credentials provider=EcsContainer error=the credentials provider was not properly configured: invalid full URI for ECS provider (URI did not refer to the loopback interface): http://169.254.170.23/v1/credentials (InvalidConfiguration(InvalidConfiguration { source: "invalid full URI for ECS provider (URI did not refer to the loopback interface): http://169.254.170.23/v1/credentials" }))

basically vector thinks it is in ECS and not on a ec2 node in a kubernetes cluster

[JeffreyVdb]

Yep, same problem on my cluster:

WARN source{component_kind="source" component_id=aws_cloudtrail component_type=aws_s3}:lazy_load_identity: aws_config::meta::credentials::chain: provider failed to provide credentials provider=EcsContainer error=the credentials provider was not properly configured: invalid full URI for ECS provider (no DNS resolver was provided. Enable `rt-tokio` or provide a `dns` resolver to the builder.): http://[fd00:ec2::23]/v1/credentials (InvalidConfiguration(InvalidConfiguration { source: "invalid full URI for ECS provider (no DNS resolver was provided. Enable `rt-tokio` or provide a `dns` resolver to the builder.): http://[fd00:ec2::23]/v1/credentials" }))

[jszwedko]

Yep, same problem on my cluster:

WARN source{component_kind="source" component_id=aws_cloudtrail component_type=aws_s3}:lazy_load_identity: aws_config::meta::credentials::chain: provider failed to provide credentials provider=EcsContainer error=the credentials provider was not properly configured: invalid full URI for ECS provider (no DNS resolver was provided. Enable `rt-tokio` or provide a `dns` resolver to the builder.): http://[fd00:ec2::23]/v1/credentials (InvalidConfiguration(InvalidConfiguration { source: "invalid full URI for ECS provider (no DNS resolver was provided. Enable `rt-tokio` or provide a `dns` resolver to the builder.): http://[fd00:ec2::23]/v1/credentials" }))

I think that is a different error that was related to a bug in an older version of Vector. Are you running latest?

[JeffreyVdb]

@jszwedko Yep, this is the container I'm running:

Image:         timberio/vector:0.41.1-alpine                                                                                                                                                  
Image ID:      docker.io/timberio/vector@sha256:501e5403e19238c9073c116fb3cbb750a9201d0271b09dae88f044534803c670

also, not unimportant to mention, this error is basically occurring in a loop it seems like. It basically spams this error and the pod I'm running starts using 100% CPU.

[jszwedko]

@jszwedko Yep, this is the container I'm running:

Image:         timberio/vector:0.41.1-alpine                                                                                                                                                  
Image ID:      docker.io/timberio/vector@sha256:501e5403e19238c9073c116fb3cbb750a9201d0271b09dae88f044534803c670

also, not unimportant to mention, this error is basically occurring in a loop it seems like. It basically spams this error and the pod I'm running starts using 100% CPU.

Thanks! This looks like a related bug to the one I was thinking about. https://github.com/vectordotdev/vector/pull/21363 should fix it.

[Denton-L]

While we're waiting for the PR to be merged, what workarounds have people been using? We set our IMDS limit to 1 and have disabled the aws-auth configmap so I can only think of either having a sidecar perform auth or using long-lived credentials.

[sknmi]

Any updates on this? Would be nice to move to AWS Pod Identity.

[jszwedko]

I still haven't been able to sort out the integration test issues upgrading the aws-config crate in https://github.com/vectordotdev/vector/pull/20663. I welcome someone else giving it a shot if they are motivated. Maybe someone else would see something that I'm missing.

[mzupan]

@jszwedko is there solid docs or instructions on how to re-create the tests failing.. i can give it a try this weekend

[jszwedko]

@jszwedko is there solid docs or instructions on how to re-create the tests failing.. i can give it a try this weekend

Great! You should be able to run make test-integration-aws locally to reproduce. It uses localstack for testing.

[pschulten]

@Denton-L I'm using https://github.com/awslabs/aws-sigv4-proxy as a workaround.

[bvalente]

Hello @jszwedko, will this be included in the next release? If so, is there a scheduled date for it? Thanks 🙂

[jszwedko]

Hello @jszwedko, will this be included in the next release? If so, is there a scheduled date for it? Thanks 🙂

Hi! Yes, it will be part of the next release going out next week. Please see https://github.com/vectordotdev/vector/blob/master/RELEASES.md for more details about the release cadance and a link to the calendar.

[jszwedko]

Closed by https://github.com/vectordotdev/vector/pull/22687