blob/s3blob SignatureDoesNotMatch When trying to read the file

[t3r-cni]

go version go1.18.3 darwin/amd64
[email protected]/blob/s3blob
OS: macOS
GOARCH="amd64"

Describe the bug

I am trying to read data from an s3 bucket using the gocloud library. I create a session using the method found here. I am able to list the keys in the bucket, but when I try and read one of them I get the following aws error:

"The request signature we calculated does not match the signature you provided. Check your key and signing method."

        os.Setenv("AWS_SDK_LOAD_CONFIG", "true")
	creds := credentials.NewSharedCredentials("/location/to/creds/file", "aws-profile")

	sess, err := session.NewSession(&aws.Config{
		Region:      aws.String("proper-aws-region"),
		Credentials: creds,
	})

        checkerr(err)
	bucket, err := s3blob.OpenBucket(ctx, sess, "my-bucket", nil)
        checkerr(err)
	defer bucket.Close()

	r, err := bucket.NewReader(ctx, "key-to-data", nil) 
        //err -> The request signature we calculated does not match the signature you provided. Check your key and signing method.

What I have tried

• aws cli: This works. So I can confirm the credentials are correct
• other aws library:

os.Setenv("AWS_SDK_LOAD_CONFIG", "true")
	creds := credentials.NewSharedCredentials("/location/to/creds/file", "aws-profile")

	sess, err := session.NewSession(&aws.Config{
		Region:      aws.String("proper-aws-region"),
		Credentials: creds,
	})
svc := s3.New(sess)
out, err := svc.GetObject(&s3.GetObjectInput{
	Bucket: aws.String("my-bucket"),
	Key:    aws.String("key-to-data"),
})
//This works... ?What?

• I have tried escaping potentially problematic characters in the credentials. I didn't expect this to fix anything because I could list the keys using bucket.List

I obviously have a few solutions to get around this issue but I would like to use the gocloud library for consistency in my code.

[vangent]

Could you try with Go CDK v0.24.0 just to see if this is a regression?

https://stackoverflow.com/q/30518899 has some tips on that error, in case any apply to you.

Does your Key have any characters that would be affected by the escaping we do? https://github.com/google/go-cloud/blob/master/blob/s3blob/s3blob.go#L895

[t3r-cni]

@vangent, I have checked v0.24.0 and v0.26.0 and both produce the same error. I have looked at the stackoverflow and tried pretty much everything on there with no success. My key does have a '/' in it but I have debugged deep into the s3 request and at the time that the request is made, the credentials (and raw credentials, both are stored) look correct.

Here is another thing I have found. If I dont use the session, it works fine:

b, err := blob.OpenBucket(ctx, "s3://my-bucket?region= proper-aws-region")
bytes, err := b.ReadAll(ctx, key)
checkError(err) //no error

But only if my credentials file looks different. This works if I don't use profiles in the credential file and only have [default]. This is problematic because I do need multiple profiles for different buckets. But it is also interesting.

Again, the previous examples do work if I have multiple profiles in my aws creds file. So something weird is happening in this:

creds := credentials.NewSharedCredentials("/location/to/creds/file", "aws-profile")
sess, err := session.NewSession(&aws.Config{
	Region:      aws.String("proper-aws-region"),
	Credentials: creds,
})

checkerr(err)
bucket, err := s3blob.OpenBucket(ctx, sess, "my-bucket", nil)
r, err := bucket.ReadAll((ctx, "key-to-data") //err

versus this:

os.Setenv("AWS_SDK_LOAD_CONFIG", "true")
creds := credentials.NewSharedCredentials("/location/to/creds/file", "aws-profile")

sess, err := session.NewSession(&aws.Config{
	Region:      aws.String("proper-aws-region"),
	Credentials: creds,
})
svc := s3.New(sess)
out, err := svc.GetObject(&s3.GetObjectInput{
	Bucket: aws.String("my-bucket"),
	Key:    aws.String("key-to-data"),
})//no err

[t3r-cni]

Also remember, that the initial code that was producing an error worked to list the keys. If I didn't read the data and I just listed the keys I could get a full list of keys, which would be weird if my credentials were bad.

[vangent]

Sorry, it's hard for me to debug this since I can't reproduce.

Can you use gohack to download Go CDK locally and debug? https://github.com/rogpeppe/gohack

I.e., you could add a `fmt.Printf, probably near here: https://github.com/google/go-cloud/blob/master/blob/s3blob/s3blob.go#L828

to see exactly what request Go CDK is making and compare it to the one you're doing. I'm guessing the error is being produced there, and would be returned a couple of lines down, but it would be good to verify that.

[t3r-cni]

@vangent, I will double check that. I might not be able to do it today but I will report back when I am able. Thanks