aws-sdk-java
aws-sdk-java copied to clipboard
aws
Generating cacheable presigned URL with S3?
My service was generating cacheable pre-signed S3 URLs using the SDK last year (2018) and from what I see, it was using S3QueryStringSigner to generate those URLs because they looks like:
https://examplebucket.s3.amazonaws.com/test2.txt?AWSAccessKeyId=AKIAEXAMPLEACCESSKEY&Signature=EXHCcBe%EXAMPLEKnz3r8O0AgEXAMPLE&Expires=1556132848
This is also the format generated by the cli: https://docs.aws.amazon.com/cli/latest/reference/s3/presign.html
The URL generated above is cacheable by the browser because I can control the expiry timestamp by passing it into s3.generatePresignedUrl().
However, at some point this year (2019), the default signer in the SDK has been changed to AWSS3V4Signer which generates non-cacheable URLs because the generated URL has a date component in it such as X-Amz-Date=20191113T001701Z
This date component was generated by calling currentTimeMillis() on the SDK clock: https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-core/src/main/java/com/amazonaws/auth/internal/AWS4SignerRequestParams.java#L129, as a result, I have no control over this behaviour.
I can see that the signer is selected using this logic: https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-s3/src/main/java/com/amazonaws/services/s3/AmazonS3Client.java#L3384
Theoretically I can make it use S3QueryStringSigner but do you recommend that? Will S3QueryStringSigner be deprecated in the future?
https://aws.amazon.com/blogs/aws/amazon-s3-update-sigv2-deprecation-period-extended-modified/
SigV2 can still be used, but it's not recommended, and is not the default.
@spfink Thanks for the response!
What about the default S3 signer, is there a sunset day for that?
And what do you suggest we do to proceed on this? Should I submit a feature request through our AWS TAM or support channel? Is there a workaround for this?
We are serving private contents to our users with an expiry of an hour. Within the hour, the content can get a lot of views. As a result, we would like to be able to cache it in the CDN and the browser.
There are no plans to remove the sigv2 signer from the SDK.
You can specify the signer to use directly through ClientConfiguration. I believe the sigv2 signer would be specified as "S3SignerType"
Thanks! Can I add a feature request for the SDK to allow overwriting the date parameter in the V4 signer? I can see that there is already the ability the do so in the code but that's only for testing at the moment.
Which method are you using for generating a presigned url?
Both the request object:
https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-s3/src/main/java/com/amazonaws/services/s3/model/GeneratePresignedUrlRequest.java#L68
And the methods in the S3 client:
https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-s3/src/main/java/com/amazonaws/services/s3/AmazonS3Client.java#L3303
Allow you to specify the expiration date.
As mentioned in the original post, the generated V4 URL has a date component in it such as X-Amz-Date=20191113T001701Z, the value of it is generated by the SDK by calling currentTimeMillis() on the SDK clock. I don't have control over that at the moment
Picking this up after a long time, I'm sorry for the long silence here @jackyjjc-canva.
As @spfink mentioned, you can specify the expiration date to GeneratePresignedUrlRequest like:
// Set the pre-signed URL to expire after one hour.
java.util.Date expiration = new java.util.Date();
long expTimeMillis = expiration.getTime();
expTimeMillis += 1000 * 60 * 60;
expiration.setTime(expTimeMillis);
GeneratePresignedUrlRequest generatePresignedUrlRequest =
new GeneratePresignedUrlRequest(BUCKET, KEY)
.withMethod(HttpMethod.GET)
.withExpiration(expiration);
and the generated url will contain both X-Amz-Date=20200408T021052Z and X-Amz-Expires=3599 (among other headers).
Please let us know if this is not what you're looking for.
@debora-ito Hi, thank you for the follow up! I think I didn't explain the use case clearly.
We use the url signer to give user access to private S3 contents. User will then use their browser to access this content via the url generated.
The flow goes like this:
- User opens a webpage in their web browser, it request an S3 object
- Our service uses S3 signer to sign the url, gets a url with
X-Amz-Date=<current-timestamp>, returns it to the user - user browser displays it.
- User refreshes the page in their web browser
- it sends the same requests to our service
- Our service uses the S3 signer to sign the url again, gets a different url with
X-Amz-Date=<new current timestamp>, returns to the user - The user's browser sees a different URL, tries to load the content again instead of using the existing cached version.
This negatively impacts our user experience since the user need to load all these large images on every page refresh due to the non-cachable S3 urls.
With GeneratePresignedUrlRequest, I can only specify the expiry and it doesn't not allow me to set the value of X-Amz-Date, so it doesn't help us in this case.
@debora-ito Hi! Just to follow up on this ticket. I have responded, do you mind removing the response-requested label?
Hello there! Does we have a solution for this issue? Or a workaround?
I'm facing the same problem here..
Same here, we can't mock the java clock as easy as it is in other languages like ruby, js, etc!
Yes, if we could specify the signing date to be pinned, for example to the most-recent time of 00:00, 06:00, 12:00, or 18:00, and the expiration to happen twelve hours from then, we could have nicely browser-cache-friendly URLs that still had the desirable security properties of being short-lived. Is there any hope of either letting us directly specify X-Amz-Date for this purpose, or perhaps adding a “signing date resolution” parameter, to ask that it be rounded to the most recent n-hour period or day, or some such thing?
@debora-ito @spfink , could you please let us know if there is any resolution for this flow ?
