Incorrect Base64 encoded x-amz-content-sha256 value

[danielcweeks]

Describe the bug

SigV4 Signing produces incorrect value for checksum header

This line in AbstractAws4Signer is producing base64 encoded values for the x-amz-content-sha256 header instead of the required "Lowercase base 16 encoding." value.

Expected Behavior

The x-amz-content-sha256 header value should be e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

Current Behavior

The x-amz-content-sha256 header value currently produced is 47DEQpj8HBSa+/TImW+5JCeuQeRkm5NMpJWZG3hSuFU= and signature verification fails

Reproduction Steps

Construct signing params

      Aws4SignerParams params =
          Aws4SignerParams.builder()
              .signingName(signingName)
              .signingRegion(signingRegion)
              .awsCredentials(credentialsProvider.resolveCredentials())
              .checksumParams(
                  SignerChecksumParams.builder()
                      .algorithm(Algorithm.SHA256)
                      .isStreamingRequest(false)
                      .checksumHeaderName(SignerConstant.X_AMZ_CONTENT_SHA256)
                      .build())
              .build();

Create a signer

Aws4Signer signer = Aws4Signer.create();

Sign and inspect the header value

SdkHttpFullRequest sdkRequest = SdkHttpFullRequest.builder()....build();
SdkHttpFullRequest signedSdkRequest = signer.sign(sdkRequest, params);

signedSdkRequest.headers().get("x-amz-content-sha256")

Possible Solution

No response

Additional Information/Context

No response

AWS Java SDK version used

master

JDK version used

Java 17

Operating System and version

OSC

[debora-ito]

@danielcweeks can you provide a self-contained code example I can use to reproduce the error? Also provide the stacktrace with the error.

[danielcweeks]

Here's a complete example:

import java.net.URI;
import org.junit.Test;
import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
import software.amazon.awssdk.auth.signer.Aws4Signer;
import software.amazon.awssdk.auth.signer.internal.SignerConstant;
import software.amazon.awssdk.auth.signer.params.Aws4SignerParams;
import software.amazon.awssdk.auth.signer.params.SignerChecksumParams;
import software.amazon.awssdk.core.checksums.Algorithm;
import software.amazon.awssdk.http.SdkHttpFullRequest;
import software.amazon.awssdk.http.SdkHttpMethod;
import software.amazon.awssdk.regions.Region;

public class TestSigV4Example {

  @Test
  public void testEmptyBodySignature() {
    Aws4SignerParams params =
        Aws4SignerParams.builder()
            .signingName("execute-api")
            .signingRegion(Region.US_WEST_2)
            .awsCredentials(AwsBasicCredentials.create("key", "secret"))
            .checksumParams(
                SignerChecksumParams.builder()
                    .algorithm(Algorithm.SHA256)
                    .isStreamingRequest(false)
                    .checksumHeaderName(SignerConstant.X_AMZ_CONTENT_SHA256)
                    .build())
            .build();

    SdkHttpFullRequest sdkRequestBuilder = SdkHttpFullRequest.builder()
        .method(SdkHttpMethod.GET)
        .protocol("https")
        .uri(URI.create("https://localhots:80/some/path"))
        .build();

    Aws4Signer signer = Aws4Signer.create();
    SdkHttpFullRequest signedSdkRequest = signer.sign(sdkRequestBuilder, params);

    String expected = "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855";
    String actual = signedSdkRequest.firstMatchingHeader(SignerConstant.X_AMZ_CONTENT_SHA256).get();

    assertThat(actual).isEqualTo(expected);
  }
}

The result:

expected: "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
 but was: "47DEQpj8HBSa+/TImW+5JCeuQeRkm5NMpJWZG3hSuFU="

[danielcweeks]

According to the SigV4 Documentation

The x-amz-content-sha256 header is required for all AWS Signature Version 4 requests. It provides a hash of the request payload. If there is no payload, you must provide the hash of an empty string.

And

HashedPayload is the hexadecimal value of the SHA256 hash of the request payload.

Hex()	Lowercase base 16 encoding.
SHA256Hash()	Secure Hash Algorithm (SHA) cryptographic hash function.

Hex(SHA256Hash() If there is no payload in the request, you compute a hash of the empty string as follows: Hex(SHA256Hash("")) The hash returns the following value: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

As you can see, the resulting value is not base 16 encoded and is not equal to the required value.

[debora-ito]

@danielcweeks Thank you for the repro code. When you say the signature verification fails, can you tell us where it fails, with which service? Can you give us more details about your use case?

Asking this because there are different uses for the X_AMZ_CONTENT_SHA256 header that are not documented in the SigV4 documentation page. I see you have a related PR made to Apache Iceberg, I just want to determine if a change is needed and if the change is in scope of the SDK.

[debora-ito]

Brought this up for discussion, the team also want to know more details about the use case.

[danielcweeks]

@danielcweeks Thank you for the repro code. When you say the signature verification fails, can you tell us where it fails, with which service? Can you give us more details about your use case?

The signature verification fails with a signed GET request to a service like API Gateway. You just need to sign and execute a request like shown above against an API Gateway endpoint and your request will fail. You can workaround this by explicitly setting the checksum to the empty string value. This bypasses the logic in the SDK because the header is already present.

Asking this because there are different uses for the X_AMZ_CONTENT_SHA256 header that are not documented in the SigV4 documentation page. I see you have a related PR made to Apache Iceberg, I just want to determine if a change is needed and if the change is in scope of the SDK.

We've worked around the issue in the way I've described above, but I would prefer to be able to rely on the SDK to correctly sign requests.

[brooks-mikkelsen]

Hey, I ran into this issue when trying to sign requests to S3 as well.

From looking at the internal implementation of the SDK, I see there are two ways to get it to add the x-amz-content-sha256 header:

1. Setting Aws4SignerParams.builder().checksumParams(...). This will produce a base-64 encoded content sha header, which causes S3 to return a 400 status code.
2. Adding the x-amz-content-sha256 header with value required to your request before signing. In this case, you don't configure Aws4SignerParams.builder().checksumParams(...). This will produce the correct base-16 encoded content sha header when Aws4Signer.sign() is called, and the resulting S3 request succeeds.

I don't see any documentation of option 2, so I'm assuming it's an internal feature that SDK consumers probably shouldn't be coding against? Or is it an official supported API of the signer?