in ecs got security token service credentials is empty.

[dybxin]

Documentation

• [X] I've gone though the API reference
• [X] I've checked AWS Forums and StackOverflow for answers
• [X] I've searched for previous similar issues and didn't find any solution

Describe the bug

in ecs cluster, i deploy services in fargate. when i use aws-sdk-go-v2 to get security token service credentials, i got an empty response result.

the response result not contains AccessKeyID 、SecretAccessKey and SessionToken.

Expected behavior

i expected got the complete response.

Current behavior

2022-01-06T17:37:39.064+08:00 | value &{ AssumeRoleProvider false 0001-01-01 00:00:00 +0000 UTC}
2022-01-06T17:39:10.810+08:00 |
err operation error S3: CreateBucket, https response error StatusCode: 400, RequestID: F3ARTKKHJ21SGJ1J, HostID: sH3s7sJOnyzMI63/VE6uFEmF9KYXKiRohUwqPaRre5DAIn/xCUCsBwRqCai6mRFc4FWGa9WFc04=, api error AuthorizationHeaderMalformed: The authorization header is malformed; a non-empty Access Key (AKID) must be provided in the credential.

Steps to Reproduce

// the roleArn is the ecsTaskExecutionRole role's arn.
roleArn = "arn:aws-cn:iam::xxxxx:role/ecsTaskExecutionRole"
region  = "cn-northwest-1"

config, err := config.LoadDefaultConfig(context.Background(), config.WithRegion(region))
if err != nil {
	panic(error)
}

stsSvc := sts.NewFromConfig(config)
creds := stscreds.NewAssumeRoleProvider(stsSvc, roleArn)
value, err := creds.Retrieve(context.TODO())
if err != nil {
	// handle error
}
fmt.Println("value", &value)
config.Credentials = aws.NewCredentialsCache(creds)

s3Cient := s3.NewFromConfig(config)

output, err := s3Cient.CreateBucket(context.TODO(), &s3.CreateBucketInput{
	Bucket:                    aws.String("images"),
	CreateBucketConfiguration: &types.CreateBucketConfiguration{LocationConstraint: 
        types.BucketLocationConstraintCnNorthwest1},
})

if err != nil {
	fmt.Println("err", err)
	var alreadyOwnedByYou *types.BucketAlreadyOwnedByYou
	var alreadyExists *types.BucketAlreadyExists
	if errors.As(err, &alreadyOwnedByYou) || errors.As(err, &alreadyExists) {
		//return
		fmt.Println("------")
	}
}
fmt.Sprintf("bucket location: %s", aws.ToString(output.Location))

Possible Solution

No response

AWS Go SDK version used

1.16.6

Compiler and Version used

go version go1.16.6 linux/amd64

Operating System and version

Amazon Linux 2 AMI (HVM) - Kernel 5.10, SSD Volume Type

[chrnorm]

I had some similar error messages using an outdated SDK version - they were solved for me by updating the SDK with go get github.com/aws/aws-sdk-go-v2. Not sure if this is helpful so I am watching this ticket too for any news on a breaking API change on AWS's side.

[dybxin]

I had some similar error messages using an outdated SDK version - they were solved for me by updating the SDK with go get github.com/aws/aws-sdk-go-v2. Not sure if this is helpful so I am watching this ticket too for any news on a breaking API change on AWS's side.

my problem is solved. in ecs fargate i not set public IP, caused by get sts svc timeout.

[KaibaLopez]

Hi @dybxin , Just to be clear, have you solved your issue or is this still a problem you are facing?

[dybxin]

Hi @dybxin , Just to be clear, have you solved your issue or is this still a problem you are facing?

yeah i solved already.

[udf2457]

@KaibaLopez @dybxin

I seem to be getting the same issue ? I think there might be a bug here ?

My code (based on the example given in the docs):

func AssumeRoleWithSts(roleARN string) (aws.Config, error) {
	cfg, err := config.LoadDefaultConfig(context.TODO())
	if err != nil {
		return aws.Config{}, err
	}
	stsSvc := sts.NewFromConfig(cfg)
	creds := stscreds.NewAssumeRoleProvider(stsSvc, roleARN)
	cfg.Credentials = aws.NewCredentialsCache(creds)
	return cfg, nil
}

Which is then passed into an S3 caller ...

func Upload(cfg aws.Config) error {
	client := s3.NewFromConfig(cfg)
etc. etc.

I get the following error:

operation error S3: ListObjectsV2, https response error StatusCode: 400, RequestID: HB4N3M8KNPXC6XTK, HostID: n3Y0V+t+0hUEz+gMGUF4Daeca2QMrDVnTjCVBOqRkhxhOTUuUSJTKqvz0XNHKKIQ68pRgRf3mjA=, api error AuthorizationHeaderMalformed: The authorization header is malformed; a non-empty Access Key (AKID) must be provided in the credential.

However, if I rely on the ~/.aws/credentials method instead, i.e.:

func LoadBasicAuthWithProfile(profile string) (aws.Config, error) {
	cfg, err := config.LoadDefaultConfig(context.TODO(), config.WithSharedConfigProfile(profile))
	if err != nil {
		return aws.Config{}, err
	}
	return cfg, nil
}

and defining role_arn and source_profile in there, it works.

Weird !

[oakad]

Recently, I managed to trigger this very error on EKS. Upon investigation, a culprit of my issue was as following:

1. Trust policy for the intended pod role had wrong service account value specified in the condition block.
2. This, in turn, caused "implicit" AssumeRoleWithWebIdentity call issued by the authentication provider to fail. This failure is not properly processed or reported by authentication provider, at least in my version of the SDK (1.16.14 base version and corresponding component versions).
3. All subsequent calls issues via the s3 client (v1.27.2) would then fail with the AKID missing error.

In short, web identity authentication provider should be modified to report errors properly and not just swallow them.