new feature: support AWS Container credential provider

[Samrose-Ahmed]

Feature Description

via AWS_CONTAINER_CREDENTIALS_RELATIVE_URI environment variable in ECS/container settings: see https://docs.aws.amazon.com/sdkref/latest/guide/feature-container-credentials.html

Problem and Solution

load credentials in ecs environment

Additional Context

No response

Are you willing to contribute to the development of this feature?

• [ ] Yes, I am willing to contribute to the development of this feature.

[Xuanwo]

Hi, @Samrose-Ahmed, does this issue related to https://github.com/apache/opendal/issues/4964?

[Samrose-Ahmed]

Yes it's the same.

[Xuanwo]

Thank you! I will redirect the discussion here.

[Xuanwo]

Hi, @Samrose-Ahmed, I expect to add this feature in our next release.

[ChillFish8]

General question regarding this, would it be a good idea to wrap the AWS SDK with the credentials provider to do this job? This has been my current workaround for this issue and might be a better long-term solution for managing all the various ways you can authenticate with AWS.

[Xuanwo]

General question regarding this, would it be a good idea to wrap the AWS SDK with the credentials provider to do this job? This has been my current workaround for this issue and might be a better long-term solution for managing all the various ways you can authenticate with AWS.

Thanks for the suggestion. We previously had a discussion here: https://github.com/apache/opendal/discussions/139.

The various methods for authentication are limited and rarely change, so we can ultimately support them all. I believe it isn't worth introducing the AWS SDK for authentication.

[matthias-Q]

I stumbled over this issue today. Here is a workaround with a start script that sets the ENV vars when running inside ECS

#!/bin/bash

CREDS_URI=${AWS_CONTAINER_CREDENTIALS_FULL_URI:-"http://169.254.170.2${AWS_CONTAINER_CREDENTIALS_RELATIVE_URI}"}
CREDS_RESPONSE=$(curl -s $CREDS_URI)

export AWS_ACCESS_KEY_ID=$(echo $CREDS_RESPONSE | jq -r '.AccessKeyId')
export AWS_SECRET_ACCESS_KEY=$(echo $CREDS_RESPONSE | jq -r '.SecretAccessKey')
export AWS_SESSION_TOKEN=$(echo $CREDS_RESPONSE | jq -r '.Token')

exec "$@"

I hope OpenDAL will support the proper Credentials Chain in the future.

[Xuanwo]

Hi, thank you @matthias-Q for this workaround. I'm currently very active in the project https://github.com/Xuanwo/reqsign, which is the crate that opendal relies on for this feature. I plan to integrate these changes in the next major version of opendal.