s3 client behaving strange while generating pre signed url

[sunilke]

Describe the bug

I am using java aws sdk and currently creating the s3 client using following manner:

AmazonS3 s3Client = new AmazonS3Client(myAwsCredentials);

I know this AmazonS3Client() is now deprecated. Now when my service that's using this s3 client comes up , I use generatePresignedUrl() using this s3Client for encrypted(AES and/or aws:kms) buckets to share the private objects(i.e GET method being used for generatePresignedUrl), it gives preSignedUrl in response that's signed using signature version 2 which obviously gives error as encrypted buckets only work with signature version 4 but when I again generate this signed url using generatePresignedUrl() then it gives me url signed with signature version 4 and that works fine as expected. So as soon as service comes up , this generatePresignedUrl() initially gives url signed with signature version 2 but then for every forthcoming requests on generatePresignedUrl() for that bucket gives url signed with signature version 4.

I even tried both of the solutions mentioned here for java sdk https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingAWSSDK.html#specify-signature-version , but none of them worked .

I am not able to understand why generatePresignedUrl() returns url signed with signature version 2 for first request on encrypted bucket as soon as service comes up and then after it starts giving url signed with signature version 4 for every forthcoming requests.

Only option I had is to start using latest way of creating s3 client , which is like this:

AmazonS3 s3Client= AmazonS3ClientBuilder.standard().withRegion(Regions.AP_SOUTH_1).withCredentials(new AWSStaticCredentialsProvider(myCredentials)).build();

This solves the above stated problem as after using this way to create s3 client, we everytime get url signed with signature version 4 from generatePresignedUrl().

But I can't right away make this change on my service as its getting used by many other services within the organization so it will take time to do this change. So until then, this deprecated way of creating s3 client will be in effect:

AmazonS3 s3Client = new AmazonS3Client(myAwsCredentials);

I want to know why this issue is happening ?

Expected behavior

signed url returned by generatePresignedUrl method should be everytime signed with signature version 4.

Current behavior

As soon as the java service that's using java aws sdk comes up , when we use generatePresignedUrl() for first time on bucket then url returned in response is signed with signature version 2 which obviously will not work on encrypted buckets but every forth coming requests to generatePresignedUrl() for that bucket returns url signed with signature version 4 which works fine as expected

So , for eg

-on first request to generatePresignedUrl() for encrypted bucket I get this url in response which is signed using signature version 2:

https://my-bucket-name.s3.amazonaws.com/%2FmyObjectKey?AWSAccessKeyId=XXX&Expires=1641358312&Signature=hvV4jizcOXLf09SAfzpKXFvkbIE%3D

• On every forthcoming requests to generatePresignedUrl() for encrypted bucket I get this type of url in response which is signed using signature version 4:

https://my-bucket-name.s3.amazonaws.com/%2FmyObjectKey?X-Amz-Algorithm\u003dAWS4-HMAC-SHA256\u0026X-Amz-Date\u003d20220104T093757Z\u0026X-Amz-SignedHeaders\u003dhost\u0026X-Amz-Expires\u003d99\u0026X-Amz-Credential\u003dXXX%2F20220104%2Fap-southeast-1%2Fs3%2Faws4_request\u0026X-Amz-Signature\u003da809e0bd4526fef489172b15b31248ce65e191b257cc321174f963e37706b6b0

Just for context: 1)This is the way s3client is being created in my java service: AmazonS3 s3Client = new AmazonS3Client(myAwsCredentials); 2)we are using GET method with generatePresignedUrl() call to download the objects

Steps to Reproduce

In java service, create s3 client using this way:

AmazonS3 s3Client = new AmazonS3Client(myAwsCredentials);

When service comes up , try using generatePresignedUrl() with GET method multiple times on encrypted(AES or aws:kms) and see difference in signature version used for returned signed urls.

Possible Solution

No response

Context

No response

AWS Java SDK version used

1.11.873

JDK version used

1.8

Operating System and version

Macos cataline version 10.15.7

[debora-ito]

@sunilke thank you for reaching out. To clarify, you are saying that when you call generatePresignedUrl() the first time the URL looks like the first link, and subsequent calls using the exact same code the URL generated is like the second link. Is this the issue?

[sunilke]

Yes , exactly and this issue occurs only when I configure s3 client in my code like this: AmazonS3 s3Client = new AmazonS3Client(myAwsCredentials);

So here new AmazonS3Client() is deprecated way as per java aws sdk.

when I configure s3 client like below then everything works fine: AmazonS3 s3Client= AmazonS3ClientBuilder.standard().withRegion(Regions.AP_SOUTH_1).withCredentials(new AWSStaticCredentialsProvider(myCredentials)).build();

But I can't make this change right away as many other services are dependent on this service. So yes please help me with issue I am facing while creating s3 client using new AmazonS3Client(myAwsCredentials);

Thanks

[debora-ito]

If the issue is happening without a change in the code is unlikely that the difference is an issue in the SDK. I could not reproduce the issue, calling generatePresignedUrl() sequentially always returns an URL like the first format.

AmazonS3 s3Client = new AmazonS3Client();

GeneratePresignedUrlRequest request = new GeneratePresignedUrlRequest(BUCKET,KEY)
                .withMethod(HttpMethod.GET)
                .withExpiration(expirationDate);

URL url = s3Client.generatePresignedUrl(request);
URL url2 = s3Client.generatePresignedUrl(request);
URL url3 = s3Client.generatePresignedUrl(request);

Are you sure you are using the same s3 client instance in every call? Is it possible that a different s3 client is being used in the subsequent calls?

On a side note: I noticed that the URLs you shared in the description contain your Access Key IDs, please rotate them as soon as possible. I edited them out of the description but they will still show in the comment history.

[sunilke]

If calling generatePresignedUrl() sequentially always returns an URL like the first format then its wrong right? because that url will not work for encrypted buckets .

Also to get signed url in second formate please try this flow: instead of directly calling generatePresignedUrl() on existing object ,when service comes up try uploading the new object on encrypted bucket and then call the generatePresignedUrl() ,it will return url signed with signature 4 I guess because thats what happening with me.

Also this access key id is sensitive? because its anyways send when we share pre signed url Anyways this ids are of test env user but If its really confidential then could you please help me deleting the issue .

[debora-ito]

If calling generatePresignedUrl() sequentially always returns an URL like the first format then its wrong right? because that url will not work for encrypted buckets .

My point is there's no change in the url format when calling generatePresignedUrl sequentially, so I cannot reproduce the behavior you are experiencing. The fact that the first format doesn't work with encrypted buckets is a different issue.

Also to get signed url in second formate please try this flow (...)

Can you provide a code sample I can use to simply copy+paste and execute? For example, I don't know what "when service comes up" means, so a self-contained reproducible code removes this type of question.

Also this access key id is sensitive?

Sharing the Access Key Id alone (without the secret key) is not a security breach, but it's not recommended. For more best practices in managing access keys you can check: https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html

[sunilke]

So basically my application is spring boot app where on startup of this application I am creating AmazonS3 bean using following way: AmazonS3 s3Client = new AmazonS3Client(myAwsCredentials); Then we have one api to get signed url for object(its very basic api that takes bucket name,key name , expiration time and return signed url in response) and implementation of this api includes only this:

GeneratePresignedUrlRequest preSignedRequest = new GeneratePresignedUrlRequest(payload.getBucketName(), payload.getKey()); preSignedRequest.setExpiration(date);

return s3Client.generatePresignedUrl(presignedUrlRequest);

Could you try setting up such application and on application startup : First upload object on s3 via any means(s3 api,etc) ,then generate signed url for this object using this application api.

Sorry if this process is troublesome but this is how my application uses s3 client

[debora-ito]

This is what my sample code is doing in a simplified way, isn't? I'm not using the API to provide bucket and key names, I'm setting them directly.

AmazonS3 s3Client = new AmazonS3Client();

GeneratePresignedUrlRequest request = new GeneratePresignedUrlRequest(BUCKET,KEY)
                .withMethod(HttpMethod.GET)
                .withExpiration(expirationDate);

URL url = s3Client.generatePresignedUrl(request);

If not, can you show what's the difference?

[sunilke]

The flow I wanted to simulate is when our sprint boot service comes up then we should be able to get signed url for keys passed on run but the way you are calling generatePresignedUrl() I guess you need to restart your service for different bucket name and key as you are not taking bucket and key but instead setting directly.

Thats what I wanted you to check that when your sprint boot app starts you should be able to call generatePresignedUrl() with dynamic bucket name and key instead of hardcoded . Then check if everytime we are getting key in first format or not.

This what happens with me , why my sprint boot app starts and I request generatePresignedUrl() by passing some existing object then url is returned in first format . Then I upload some new object using s3 post policy and request generatePresignedUrl() for that new object ,the url that's return is now in second format(sigv4) and also when I again request generatePresignedUrl() for old object then I get url in second format(sigv4) for which I previously got url in first format(sigv2). Here you see all of the operations were requested on run but if your code also simulate this flow of requesting generatePresignedUrl() on run with dynamic values then we can further discuss the issue of getting url with first format(sigv2) from s3Client.

[debora-ito]

Dynamically defining bucket and key values in GeneratePresignedUrlRequest() should not change the format of the generated url. I understand your use case as you describe it, but to investigate further I must insist you please provide a code sample we can use to reproduce. http://sscce.org/

[github-actions[bot]]

It looks like this issue has not been active for more than five days. In the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please add a comment to prevent automatic closure, or if the issue is already closed please feel free to reopen it.

[sunilke]

My project is spring boot application is using rest apis but not with @RestController or @Controller. , we are using some internal mechanisms to create rest apis,etc. I can describe you the flow and then you can write the rest apis for same, its very basic and easy flow. So we have one api that send out s3 signed post policy(post policy) for specific bucket , its response looks like this:

{
    "postAction": "https://bucketName.s3.amazonaws.com",
    "acl": "public-read-write",
    "key": "f59414aa-c303-4a04-bfa8-fa47585d3b8c",
    "policy": "eyJleHBpcmF0aW9uIjoiMjAyMi0wMy0xN1QwNjowNDo1Mi40MzhaIiwiY29uZGl0aW9ucyI6W3siYWNsIjoicHVibGljLXJlYWQtd3JpdGUifSx7ImJ1Y2tldCI6InpldGEtczMtYnVja2V0LXRlbXBvcmFyeS1wcmVwcm9kIn0seyJrZXkiOiJmNTk0MTRhYS1jMzAzLTRhMDQtYmZhOC1mYTQ3NTg1ZDNiOGMifSx7InN1Y2Nlc3NfYWN0aW9uX3N0YXR1cyI6IjIwMSJ9LHsieC1hbXotYWxnb3JpdGhtIjoiQVdTNC1ITUFDLVNIQTI1NiJ9LHsieC1hbXotY3JlZGVud",
    "successActionStatus": "201",
    "xAmzAlgorithm": "AWS4-HMAC-SHA256",
    "xAmzCredential": "iam_user_access_key/20220317/bucket_region/s3/aws4_request",
    "xAmzDate": "20220317T000000Z",
    "xAmzSignature": "amzSignature",
    "headers": {}
}

Now then please upload any file using above signed policy , we can use following html file to upload object using above signed post policy:

<html>
  <head>
    
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
    
  </head>
  <body>

  <form action="http://sigv4examplebucket.s3.amazonaws.com/" method="post" enctype="multipart/form-data">
    Key to upload: 
    <input type="input"  name="key" value="user/user1/${filename}" /><br />
    <input type="hidden" name="acl" value="public-read" />
    <input type="hidden" name="success_action_redirect" value="http://sigv4examplebucket.s3.amazonaws.com/successful_upload.html" />
    Content-Type: 
    <input type="input"  name="Content-Type" value="image/jpeg" /><br />
    <input type="hidden" name="x-amz-meta-uuid" value="14365123651274" /> 
    <input type="hidden" name="x-amz-server-side-encryption" value="AES256" /> 
    <input type="text"   name="X-Amz-Credential" value="AKIAIOSFODNN7EXAMPLE/20151229/us-east-1/s3/aws4_request" />
    <input type="text"   name="X-Amz-Algorithm" value="AWS4-HMAC-SHA256" />
    <input type="text"   name="X-Amz-Date" value="20151229T000000Z" />

    Tags for File: 
    <input type="input"  name="x-amz-meta-tag" value="" /><br />
    <input type="hidden" name="Policy" value='<Base64-encoded policy string>' />
    <input type="hidden" name="X-Amz-Signature" value="<signature-value>" />
    File: 
    <input type="file"   name="file" /> <br />
    <!-- The elements after this will be ignored -->
    <input type="submit" name="submit" value="Upload to Amazon S3" />
  </form>
  
</html>

Now we have second api that copies the above uploaded object to some other bucket, so this api takes source bucket , key , etag,destination bucket and then s3 copy object is performed :

CopyObjectRequest copyObjectRequest = new CopyObjectRequest(sourceS3Bucket, sourceKey,
      destinationBucket, destinationKey);
transferManager.copy(copyObjectRequest, (transfer, transferState) -> {
          if (transfer.isDone()) {
            if (transferState.equals(Transfer.TransferState.Completed)) {
              future.complete(new AssetLocation(
                String.format(S3_URL_FORMAT, destinationBucket, destinationKey)));
            } else {
              future.completeExceptionally(new S3TransferException(
                "Transfer to permanent storage failed: " + transferState.toString()));
            }
          }
        });

Once copy object is done , we then use 3rd api to get signed url of object copied to the destination bucket using above api. This signed url api takes bucket name, key as query params , then it generates and returns the signed s3 url:

This Api Implementation:

GeneratePresignedUrlRequest preSignedRequest = new GeneratePresignedUrlRequest(bucketName, key);
    preSignedRequest.setExpiration(date);
return s3Client.generatePresignedUrl(presignedUrlRequest);

This what happens with me , why my sprint boot app starts and I request 3rd api to get presigned url for object by passing some existing object then url is returned in first format . Then I upload some new object using s3 post policy thats returned by 1st api ,then I perform copy object using 2nd api for this uploaded object to move it to some other bucket and once that is done I use 3rd api to get presigned url for this object that I copied to some destination bucket ,the url that's return is now in second format(sigv4).

Please see if this helps.

[sunilke]

@debora-ito any updates for this?

[debora-ito]

Hi @sunilke, still have no idea what can be causing this.

One way to make the S3Client to use a signer that is not the default sigV4 is to explicitly set withSignerOverride() in the S3Client ClientConfiguration, which you don't seem to be using. Maybe the s3Client is being modified after it was instantiated. This is one of the reasons why new AmazonS3Client() was deprecated in favor of builders, objects created with builders are immutable.

Sorry, I don't have any other suggestions besides changing the code to use AmazonS3ClientBuilder, which you've confirmed it works as expected.

[sunilke]

Thanks @debora-ito for your inputs , will be using AmazonS3ClientBuilder only going forward .

[github-actions[bot]]

COMMENT VISIBILITY WARNING

Comments on closed issues are hard for our team to see. If you need more assistance, please open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.