Allow custom HTTP headers

[bep]

Describe the bug

I'm in the process of upgrading https://github.com/bep/s3deploy/ from v1 to v2 of this SDK. s3deploy allows the user to set HTTP headers (e.g. Cache-Control, Content-Security-Policy), but I have so far not been successful doing this with aws-sdk-go-v2.

Here is a summary of my investigations:

• There are some predefined fields for some headers like Cache-Control and Content-Type, which gets set correctly.
• Any values put in PutObjectInput.Metadata will get prefixed, e.g. x-amz-meta-content-security-policy.
• I have tried to add these header via a build middleware, and that works for some keys (e.g Content-Encoding), but not for others (e.g Content-Security-Policy). Also, any value for Cache-Control and Content-Type set gets dropped, even if they're not set in PutObjectInput.CacheControl etc.)

Expected Behavior

I expected to find a reasonably simple way to store any valid HTTP header.

Current Behavior

See the above.

Reproduction Steps

See above.

Possible Solution

I think it would be good if you could somehow give the caller a way to separate metadata (which you then prefix with x-amz-meta-) and HTTP headers (which you then just leave as is). E.g. by introducing a new Headers field on the input object. You then also must make sure that zero values in ContentType e.g. doesn't overwrite keys in Headers map.

Additional Information/Context

No response

AWS Go SDK V2 Module Versions Used

github.com/bep/s3deploy/v2 github.com/aws/[email protected]
github.com/bep/s3deploy/v2 github.com/aws/aws-sdk-go-v2/[email protected]
github.com/bep/s3deploy/v2 github.com/aws/aws-sdk-go-v2/service/[email protected]
github.com/bep/s3deploy/v2 github.com/aws/aws-sdk-go-v2/service/[email protected]
github.com/aws/[email protected] github.com/aws/[email protected]

Compiler and Version used

go version go1.19 darwin/arm64

Operating System and version

Darwin bep-mac14.local 21.4.0 Darwin Kernel Version 21.4.0: Fri Mar 18 00:46:32 PDT 2022; root:xnu-8020.101.4~15/RELEASE_ARM64_T6000 arm64

[RanVaknin]

Hi @bep ,

I'm not sure how exactly you are attempting to set the headers because you didn't provide your code, but here is a code example of how to set custom headers with functional options on the operation level:

func main() {
	cfg, err := config.LoadDefaultConfig(context.TODO(), config.WithRegion("us-east-1"), config.WithClientLogMode(aws.LogRequestWithBody))
	if err != nil {
		log.Fatalf("unable to load SDK config, %v", err)
	}

	client := s3.NewFromConfig(cfg)

	out, _ := client.GetObject(context.Background(), &s3.GetObjectInput{
		Bucket: aws.String(myBucket),
		Key:    aws.String(myKey),
	}, func(options *s3.Options) {
		options.APIOptions = []func(*middleware.Stack) error{
			smithyhttp.AddHeaderValue("X-Foo-Header", "Baz-Value"),
		}
	})
}
// more code...

Raw HTTP request :

SDK 2022/10/04 15:09:32 DEBUG Request
GET /blah?x-id=GetObject HTTP/1.1
Host: myBucket.s3.us-east-1.amazonaws.com
User-Agent: Redacted
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: Redacted
Amz-Sdk-Request: attempt=1; max=3
Authorization: Redacted, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date;x-custom-header, Signature=Redacted
X-Amz-Content-Sha256: Redacted
X-Amz-Date: 20221004T220932Z
X-Foo-Header: Baz-Value

Let me know if that helps.

Thanks, Ran~

[bep]

@RanVaknin To be more precise:

I'm talking about PutObject and HTTP headers stored as Metadata in S3 to be used as HTTP headers when hosting a bucket as a static site.

I tested your approach, and I don't see any of the headers in the S3 object (which I guess isn't very surprising):

I have also tested with a middleware func like this:

var applyHeaders = middleware.BuildMiddlewareFunc("ApplyHeaders", func(
	ctx context.Context, in middleware.BuildInput, next middleware.BuildHandler,
) (
	middleware.BuildOutput, middleware.Metadata, error,
) {
	switch v := in.Request.(type) {
	case *smithyhttp.Request:
		headers := headHeadersFromContext(ctx)
		for key, value := range headers {
			v.Header.Add(key, value)
		}
	}
	return next.HandleBuild(ctx, in)

})

This works in the sense that the headers is stored in Metadata, but they are prefixed with x-amz-meta-, which makes them useless as HTTP headers (e.g. Content-Security-Policy).

[RanVaknin]

Hi @bep ,

Is what you are trying to do documented in any S3 docs? Because I'm pretty sure S3 doesn't accept arbitrary headers. Their API is expecting that prefix as shown in the Console:

You are able to set some headers in input params.

_, err = client.PutObject(context.Background(), &s3.PutObjectInput{
		Bucket: aws.String(myBucket),
		Key:    aws.String(myKey),
		CacheControl: aws.String("foo"),
		ContentType:  aws.String("bazbar"),
})

Thanks, Ran~

[bep]

OK, now I feel stupid, I always assumed that this should work -- because how the old API (v2) worked -- but testing it now I see that headers that's not in the "system defined list". Sorry for the noise...

[github-actions[bot]]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.

[RanVaknin]

No worries :)

Thanks for asking questions and engaging in the community!

All the best, Ran~

[github-actions[bot]]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.