aws
Support `STREAMING-AWS4-HMAC-SHA256-PAYLOAD`
Describe the feature
Hi, I'd like to upload a signed body payload without having the actual body upfront (I do know the length up-front but that's it). In theory I should be using STREAMING-AWS4-HMAC-SHA256-PAYLOAD but I can't figure out how to use the SDK to do that. Is this feature not implemented? If not, is there a simple middleware layer I can install to do this?
Use Case
I have some middleware server code responsible for uploading large files to S3. I don't want to have to buffer the file in memory / disk because I'm handling a lot of concurrent requests. I do want to have some confidence that Amazon is validating the signature of the upload cryptographically. I'd like to send the data using STREAMING-AWS4-HMAC-SHA256-PAYLOAD but I don't know how to get the library to do that.
Proposed Solution
No response
Other Information
No response
Acknowledgements
- [ ] I may be able to implement this feature request
- [ ] This feature might incur a breaking change
SDK version used
3.145.0
Environment details (OS name and version, etc.)
Arch Linux
Hi @vlovich ,
Thanks for opening this issue. I guess I'm not sure how you want to calculate the signature if you do not know the body? The signing process takes into account the actual body so I don't see a way to do what you are asking.
Am I misunderstanding your request?
Thansk, Ran~
Have you seen streaming sigv4? https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming.html
The Authorization signature just says "I'm going to stream the body in a streaming sigv4 fashion" (STREAMING-AWS4-HMAC-SHA256-PAYLOAD). The body is sent in chunks that are themselves cryptographically signed.
For example, rather than including the sha256 of the body upfront (& thus needing to sha256 the entire file upfront), I can still get a signed upload for a 5 GiB file while only needing to have a small in-memory buffer for computing the signature (smaller latency too although in my use-case I don't have the full body upfront and have limited working RAM compared with the size of files I'm processing).
This is done, for example, by buffering 256 kib of body, compute the signature, send the chunk. The signature for each chunk is ratcheted (i.e. signature for chunk 2 is signature for chunk 1 + hash of empty string + hash of payload in chunk 2). That way you don't need to precompute the hash upfront but you still get a cryptographically signed upload.
