`XAmzContentSHA256Mismatch` on CockroachDB Backups

[GodTamIt]

Expected Behavior

I expect this shouldn't fail. This doesn't fail on Cloudflare R2, for example.

Current Behavior

You'll see this error:

closing object: upload failed: XAmzContentSHA256Mismatch: operation error S3: PutObject, https response error StatusCode: 400, RequestID: 1863628D0C2073C9, HostID: , api error XAmzContentSHA256Mismatch: The provided 'x-amz-content-sha256' header does not match what was computed.

Possible Solution

I think this might be an error in how checksums are handled. It may be related to some of the solutions to mitigate https://github.com/cockroachdb/cockroach/issues/115190 as well, which exposes an uncommon code path in the S3 compatibility layer.

Steps to Reproduce

Try setting up Storj as the destination bucket for CockroachDB backups.

Something along the lines of:

CREATE SCHEDULE
  storj_backup
FOR BACKUP INTO
  's3://cockroachdb-backups?AWS_ACCESS_KEY_ID=<redacted>&AWS_SECRET_ACCESS_KEY=<redacted>&AWS_ENDPOINT=https://gateway.storjshare.io&AWS_REGION=storj'
WITH revision_history
RECURRING
  '0 2,19 * * *'
FULL BACKUP
  '@weekly'
WITH SCHEDULE OPTIONS
  first_run = 'now'
;

[GodTamIt]

I've posted https://github.com/cockroachdb/cockroach/issues/153195 since I think this might be a CockroachDB issue but I'm not 100% sure.

As an aside, I wonder if this is a 0 byte file issue?

[GodTamIt]

This is related to: https://github.com/storj/gateway-st/issues/91

I posted a change in CockroachDB to work around this but this'll break a lot of clients since the defaults have changed.