Cortex Ingester S3 upload fails with `x-amz-server-side-encryption` header error

[mkieweg]

Describe the bug Uploading chunks to S3 fails with the error upload s3 object: x-amz-server-side-encryption header is not supported for this operation.

This happens predominantly when calling the ingester's /shutdown endpoint, but also during regular writes to S3.

The behaviour leads to unhealthy ingesters in the ring. This is breaking the auto scaling functionality as scaling down almost always leads to too many unhealthy ingesters in the ring.

To Reproduce Steps to reproduce the behavior:

1. Start Cortex (both 1.15.2 and 1.15.3)
2. Grep logs for x-amz-server-side-encryption

Expected behavior uploads to S3 work without errors

Environment:

• Infrastructure: EKS
• Deployment tool: Helm (Chart version v2.1.0)

Additional Context Logs:

{"caller":"bucket_client.go:124","err":"upload s3 object: x-amz-server-side-encryption header is not supported for this operation.","level":"error","msg":"bucket operation fail after retries","operation":"Upload fake/01HEBPMH7HW704NDCJPHEF552S/chunks/000001","ts":"2023-11-08T11:23:37.350644133Z"}
{"block":"01HEBPMH7HW704NDCJPHEF552S","caller":"shipper.go:307","err":"upload chunks: upload file /data/tsdb/fake/thanos/upload/01HEBPMH7HW704NDCJPHEF552S/chunks/000001 as 01HEBPMH7HW704NDCJPHEF552S/chunks/000001: upload s3 object: x-amz-server-side-encryption header is not supported for this operation.","level":"error","msg":"shipping failed","org_id":"fake","ts":"2023-11-08T11:23:37.380874401Z"}

S3 Config:

blocks_storage:
  backend: s3
  s3:
    endpoint: <endpoint name>
    region: ""
    bucket_name: <bucket name>
    secret_access_key: ""
    access_key_id: ""
    insecure: false
    signature_version: v4
    bucket_lookup_type: auto
    sse:
      type: SSE-S3
      kms_key_id: ""
      kms_encryption_context: ""
    http:
      idle_conn_timeout: 1m30s
      response_header_timeout: 2m0s
      insecure_skip_verify: false
      tls_handshake_timeout: 10s
      expect_continue_timeout: 1s
      max_idle_connections: 100
      max_idle_connections_per_host: 100
      max_connections_per_host: 0

[mkieweg]

Explicitly omitting the SSE-S3 type value mitigates the issue on AWS S3 - which is okay as all S3 buckets on S3 have SSE-S3 enabled by default now.

[alanprot]

I think if you define SSE-S3 u must define the key. Can you try that?

[mkieweg]

With SSE-S3 the key is managed by AWS so there is no key to define.