aws-sdk-go-v2 icon indicating copy to clipboard operation
aws-sdk-go-v2 copied to clipboard
verified publisher icon aws

Can't use `optFn` to set region for S3 Express operations (credentials issue)

Open stefansundin opened this issue 1 year ago • 1 comments

Acknowledgements

  • [X] I have searched (https://github.com/aws/aws-sdk/issues?q=is%3Aissue) for past instances of this issue
  • [X] I have verified all of my SDK modules are up-to-date (you can perform a bulk update with go get -u github.com/aws/aws-sdk-go-v2/...)

Describe the bug

S3 Express buckets require credentials that are implicitly retrieved with the CreateSession API operation. However, if the region is set using an optFn, that region is not propagated to the CreateSession call, and results in this error:

operation error S3: GetObject, get identity: get credentials: operation error S3: CreateSession, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Get "https://my-test-bucket--usw2-az1--x-s3.s3express-usw2-az1.us-west-1.amazonaws.com/?session=": dial tcp: lookup my-test-bucket--usw2-az1--x-s3.s3express-usw2-az1.us-west-1.amazonaws.com: no such host

In this case my S3 client was initialized with the us-west-1 region, but I'm attempting to set the region to us-west-2 in an optFn, but this is not respected in the CreateSession call.

Expected Behavior

I expect optFn functions to be forwarded to the CreateSession operation.

Current Behavior

optFn functions do not affect S3 Express credentials fetching.

Reproduction Steps

package main

import (
	"context"
	"fmt"
	"os"

	"github.com/aws/aws-sdk-go-v2/aws"
	"github.com/aws/aws-sdk-go-v2/config"
	"github.com/aws/aws-sdk-go-v2/service/s3"
)

func main() {
	bucket := "my-test-bucket--usw2-az1--x-s3" // this S3 Express bucket is in us-west-2
	key := "test"
	initialRegion := "us-west-1" // incorrect region for my S3 Express bucket
	debug := true

	// I want to set the correct region using an optFn when making the GetObject call:
	setRegionFn := func(o *s3.Options) {
		o.Region = "us-west-2"
	}

	// Initialize the AWS SDK
	cfg, err := config.LoadDefaultConfig(
		context.TODO(),
		config.WithRegion(initialRegion),
		func(o *config.LoadOptions) error {
			if debug {
				var lm aws.ClientLogMode = aws.LogRequest | aws.LogResponse
				o.ClientLogMode = &lm
			}
			return nil
		},
	)
	if err != nil {
		fmt.Fprintln(os.Stderr, err)
		os.Exit(1)
	}

	client := s3.NewFromConfig(cfg)

	obj, err := client.GetObject(context.TODO(), &s3.GetObjectInput{
		Bucket: aws.String(bucket),
		Key:    aws.String(key),
	}, setRegionFn) // <-- optFn used here

	if err != nil {
		fmt.Fprintln(os.Stderr, err)
		os.Exit(1)
	}

	fmt.Fprintf(os.Stderr, "GetObject ContentLength: %d\n", aws.ToInt64(obj.ContentLength))
}

Possible Solution

Apply optFn functions first, and then clone the resulting client options before calling CreateSession?

Additional Information/Context

No response

AWS Go SDK V2 Module Versions Used

example github.com/aws/[email protected]
example github.com/aws/aws-sdk-go-v2/aws/protocol/[email protected]
example github.com/aws/aws-sdk-go-v2/[email protected]
example github.com/aws/aws-sdk-go-v2/[email protected]
example github.com/aws/aws-sdk-go-v2/feature/ec2/[email protected]
example github.com/aws/aws-sdk-go-v2/internal/[email protected]
example github.com/aws/aws-sdk-go-v2/internal/endpoints/[email protected]
example github.com/aws/aws-sdk-go-v2/internal/[email protected]
example github.com/aws/aws-sdk-go-v2/internal/[email protected]
example github.com/aws/aws-sdk-go-v2/service/internal/[email protected]
example github.com/aws/aws-sdk-go-v2/service/internal/[email protected]
example github.com/aws/aws-sdk-go-v2/service/internal/[email protected]
example github.com/aws/aws-sdk-go-v2/service/internal/[email protected]
example github.com/aws/aws-sdk-go-v2/service/[email protected]
example github.com/aws/aws-sdk-go-v2/service/[email protected]
example github.com/aws/aws-sdk-go-v2/service/[email protected]
example github.com/aws/aws-sdk-go-v2/service/[email protected]
example github.com/aws/[email protected]
example [email protected]
github.com/aws/[email protected] github.com/aws/[email protected]
github.com/aws/[email protected] github.com/jmespath/[email protected]
github.com/aws/[email protected] [email protected]
github.com/aws/aws-sdk-go-v2/aws/protocol/[email protected] github.com/aws/[email protected]
github.com/aws/aws-sdk-go-v2/aws/protocol/[email protected] [email protected]
github.com/aws/aws-sdk-go-v2/[email protected] github.com/aws/[email protected]
github.com/aws/aws-sdk-go-v2/[email protected] github.com/aws/aws-sdk-go-v2/[email protected]
github.com/aws/aws-sdk-go-v2/[email protected] github.com/aws/aws-sdk-go-v2/feature/ec2/[email protected]
github.com/aws/aws-sdk-go-v2/[email protected] github.com/aws/aws-sdk-go-v2/internal/[email protected]
github.com/aws/aws-sdk-go-v2/[email protected] github.com/aws/aws-sdk-go-v2/service/[email protected]
github.com/aws/aws-sdk-go-v2/[email protected] github.com/aws/aws-sdk-go-v2/service/[email protected]
github.com/aws/aws-sdk-go-v2/[email protected] github.com/aws/aws-sdk-go-v2/service/[email protected]
github.com/aws/aws-sdk-go-v2/[email protected] github.com/aws/[email protected]
github.com/aws/aws-sdk-go-v2/[email protected] github.com/aws/aws-sdk-go-v2/internal/[email protected]
github.com/aws/aws-sdk-go-v2/[email protected] github.com/aws/aws-sdk-go-v2/internal/endpoints/[email protected]
github.com/aws/aws-sdk-go-v2/[email protected] github.com/aws/aws-sdk-go-v2/service/internal/[email protected]
github.com/aws/aws-sdk-go-v2/[email protected] github.com/aws/aws-sdk-go-v2/service/internal/[email protected]
github.com/aws/aws-sdk-go-v2/[email protected] [email protected]
github.com/aws/aws-sdk-go-v2/[email protected] github.com/aws/[email protected]
github.com/aws/aws-sdk-go-v2/[email protected] github.com/aws/aws-sdk-go-v2/feature/ec2/[email protected]
github.com/aws/aws-sdk-go-v2/[email protected] github.com/aws/aws-sdk-go-v2/service/[email protected]
github.com/aws/aws-sdk-go-v2/[email protected] github.com/aws/aws-sdk-go-v2/service/[email protected]
github.com/aws/aws-sdk-go-v2/[email protected] github.com/aws/aws-sdk-go-v2/service/[email protected]
github.com/aws/aws-sdk-go-v2/[email protected] github.com/aws/[email protected]
github.com/aws/aws-sdk-go-v2/[email protected] github.com/aws/aws-sdk-go-v2/internal/[email protected]
github.com/aws/aws-sdk-go-v2/[email protected] github.com/aws/aws-sdk-go-v2/internal/endpoints/[email protected]
github.com/aws/aws-sdk-go-v2/[email protected] github.com/aws/aws-sdk-go-v2/service/internal/[email protected]
github.com/aws/aws-sdk-go-v2/[email protected] github.com/aws/aws-sdk-go-v2/service/internal/[email protected]
github.com/aws/aws-sdk-go-v2/[email protected] [email protected]
github.com/aws/aws-sdk-go-v2/feature/ec2/[email protected] github.com/aws/[email protected]
github.com/aws/aws-sdk-go-v2/feature/ec2/[email protected] github.com/aws/[email protected]
github.com/aws/aws-sdk-go-v2/feature/ec2/[email protected] [email protected]
github.com/aws/aws-sdk-go-v2/internal/[email protected] github.com/aws/[email protected]
github.com/aws/aws-sdk-go-v2/internal/[email protected] github.com/aws/[email protected]
github.com/aws/aws-sdk-go-v2/internal/[email protected] [email protected]
github.com/aws/aws-sdk-go-v2/internal/endpoints/[email protected] github.com/aws/[email protected]
github.com/aws/aws-sdk-go-v2/internal/endpoints/[email protected] github.com/aws/[email protected]
github.com/aws/aws-sdk-go-v2/internal/endpoints/[email protected] [email protected]
github.com/aws/aws-sdk-go-v2/internal/[email protected] [email protected]
github.com/aws/aws-sdk-go-v2/internal/[email protected] github.com/aws/[email protected]
github.com/aws/aws-sdk-go-v2/internal/[email protected] github.com/aws/[email protected]
github.com/aws/aws-sdk-go-v2/internal/[email protected] [email protected]
github.com/aws/aws-sdk-go-v2/service/internal/[email protected] github.com/aws/[email protected]
github.com/aws/aws-sdk-go-v2/service/internal/[email protected] [email protected]
github.com/aws/aws-sdk-go-v2/service/internal/[email protected] github.com/aws/[email protected]
github.com/aws/aws-sdk-go-v2/service/internal/[email protected] github.com/aws/aws-sdk-go-v2/service/internal/[email protected]
github.com/aws/aws-sdk-go-v2/service/internal/[email protected] github.com/aws/[email protected]
github.com/aws/aws-sdk-go-v2/service/internal/[email protected] [email protected]
github.com/aws/aws-sdk-go-v2/service/internal/[email protected] github.com/aws/[email protected]
github.com/aws/aws-sdk-go-v2/service/internal/[email protected] github.com/aws/[email protected]
github.com/aws/aws-sdk-go-v2/service/internal/[email protected] [email protected]
github.com/aws/aws-sdk-go-v2/service/internal/[email protected] github.com/aws/[email protected]
github.com/aws/aws-sdk-go-v2/service/internal/[email protected] github.com/aws/[email protected]
github.com/aws/aws-sdk-go-v2/service/internal/[email protected] [email protected]
github.com/aws/aws-sdk-go-v2/service/[email protected] github.com/aws/[email protected]
github.com/aws/aws-sdk-go-v2/service/[email protected] github.com/aws/aws-sdk-go-v2/aws/protocol/[email protected]
github.com/aws/aws-sdk-go-v2/service/[email protected] github.com/aws/aws-sdk-go-v2/internal/[email protected]
github.com/aws/aws-sdk-go-v2/service/[email protected] github.com/aws/aws-sdk-go-v2/internal/endpoints/[email protected]
github.com/aws/aws-sdk-go-v2/service/[email protected] github.com/aws/aws-sdk-go-v2/internal/[email protected]
github.com/aws/aws-sdk-go-v2/service/[email protected] github.com/aws/aws-sdk-go-v2/service/internal/[email protected]
github.com/aws/aws-sdk-go-v2/service/[email protected] github.com/aws/aws-sdk-go-v2/service/internal/[email protected]
github.com/aws/aws-sdk-go-v2/service/[email protected] github.com/aws/aws-sdk-go-v2/service/internal/[email protected]
github.com/aws/aws-sdk-go-v2/service/[email protected] github.com/aws/aws-sdk-go-v2/service/internal/[email protected]
github.com/aws/aws-sdk-go-v2/service/[email protected] github.com/aws/[email protected]
github.com/aws/aws-sdk-go-v2/service/[email protected] [email protected]
github.com/aws/aws-sdk-go-v2/service/[email protected] github.com/aws/[email protected]
github.com/aws/aws-sdk-go-v2/service/[email protected] github.com/aws/aws-sdk-go-v2/internal/[email protected]
github.com/aws/aws-sdk-go-v2/service/[email protected] github.com/aws/aws-sdk-go-v2/internal/endpoints/[email protected]
github.com/aws/aws-sdk-go-v2/service/[email protected] github.com/aws/[email protected]
github.com/aws/aws-sdk-go-v2/service/[email protected] [email protected]
github.com/aws/aws-sdk-go-v2/service/[email protected] github.com/aws/[email protected]
github.com/aws/aws-sdk-go-v2/service/[email protected] github.com/aws/aws-sdk-go-v2/internal/[email protected]
github.com/aws/aws-sdk-go-v2/service/[email protected] github.com/aws/aws-sdk-go-v2/internal/endpoints/[email protected]
github.com/aws/aws-sdk-go-v2/service/[email protected] github.com/aws/[email protected]
github.com/aws/aws-sdk-go-v2/service/[email protected] [email protected]
github.com/aws/aws-sdk-go-v2/service/[email protected] github.com/aws/[email protected]
github.com/aws/aws-sdk-go-v2/service/[email protected] github.com/aws/aws-sdk-go-v2/internal/[email protected]
github.com/aws/aws-sdk-go-v2/service/[email protected] github.com/aws/aws-sdk-go-v2/internal/endpoints/[email protected]
github.com/aws/aws-sdk-go-v2/service/[email protected] github.com/aws/aws-sdk-go-v2/service/internal/[email protected]
github.com/aws/aws-sdk-go-v2/service/[email protected] github.com/aws/aws-sdk-go-v2/service/internal/[email protected]
github.com/aws/aws-sdk-go-v2/service/[email protected] github.com/aws/[email protected]
github.com/aws/aws-sdk-go-v2/service/[email protected] [email protected]
github.com/aws/[email protected] [email protected]
[email protected] [email protected]

Compiler and Version used

go version go1.23.1 darwin/arm64

Operating System and version

macOS 14.6.1

stefansundin avatar Sep 06 '24 18:09 stefansundin

Agreed. Workaround for the time being is to have multiple clients per region / config.

lucix-aws avatar Sep 06 '24 19:09 lucix-aws