terraform
terraform copied to clipboard
hashicorp
Error: `SignatureDoesNotMatch` only when using Ceph S3 running behind a cloudflare tunnel
Terraform Version
Terraform v1.10.4
on darwin_arm64
+ provider registry.terraform.io/cloudflare/cloudflare v4.51.0
+ provider registry.terraform.io/hashicorp/kubernetes v2.35.1
+ provider registry.terraform.io/hashicorp/random v3.6.3
Terraform Configuration Files
terraform {
backend "s3" {
bucket = "<redacted>"
key = "<redacted>/terraform.tfstate"
region = "default"
profile = "Terraform"
skip_region_validation = true
skip_credentials_validation = true
skip_metadata_api_check = true
skip_requesting_account_id = true
use_path_style = true
skip_s3_checksum = true
endpoints = {
# s3 = "https://<redacted>.arkobasu.space" <-- This doesn't work
s3 = "http://192.168.5.81:80" <-- This works
}
}
required_providers {
random = {
source = "hashicorp/random"
}
cloudflare = {
source = "cloudflare/cloudflare"
version = "~> 4"
}
kubernetes = {
source = "hashicorp/kubernetes"
version = "2.35.1"
}
}
}
provider "cloudflare" {
api_token = var.cf_api_token
}
provider "kubernetes" {
config_path = "~/.kube/config"
}
Debug Output
2025-02-03T00:37:25.557-0600 [DEBUG] backend-s3: HTTP Request Sent: aws.region=default aws.s3.bucket=<redacted-bucket-name> rpc.method=ListObjectsV2 rpc.service=S3 rpc.system=aws-api tf_aws.custom_endpoint=true tf_aws.sdk=aws-sdk-go-v2 tf_backend.operation=Workspaces tf_backend.req_id=05ad803a-067d-c682-41aa-fab081edf53a tf_backend.s3.bucket=<redacted-bucket-name> tf_backend.workspace-prefix=env:/ http.request.header.authorization="AWS4-HMAC-SHA256 Credential=<redacted>/20250203/default/s3/aws4_request, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date, Signature=*****" http.request.header.x_amz_content_sha256=<redacted> http.request.header.amz_sdk_request="attempt=1; max=5" http.request.header.x_amz_date=20250203T063725Z http.url="https://<redacted-record>.arkobasu.space/<redacted-bucket-name>?list-type=2&max-keys=1000&prefix=env%3A%2F" http.user_agent="APN/1.0 HashiCorp/1.0 Terraform/1.10.4 (+https://www.terraform.io) m/C aws-sdk-go-v2/1.31.0 os/macos lang/go#1.23.3 md/GOOS#darwin md/GOARCH#arm64 api/s3#1.63.0" http.request.header.amz_sdk_invocation_id=8c5bcb89-da72-4cab-8227-74a9974c963d http.request.header.accept_encoding=identity http.request.body="" http.method=GET net.peer.name=<redacted-record>.arkobasu.space
2025-02-03T00:37:25.736-0600 [DEBUG] backend-s3: HTTP Response Received: aws.region=default aws.s3.bucket=<redacted-bucket-name> rpc.method=ListObjectsV2 rpc.service=S3 rpc.system=aws-api tf_aws.custom_endpoint=true tf_aws.sdk=aws-sdk-go-v2 tf_backend.operation=Workspaces tf_backend.req_id=05ad803a-067d-c682-41aa-fab081edf53a tf_backend.s3.bucket=<redacted-bucket-name> tf_backend.workspace-prefix=env:/ http.response.header.cf_cache_status=DYNAMIC http.response.header.x_amz_request_id=tx000004599d10cd4712119-0067a06425-1192799-default http.response.header.server=cloudflare http.duration=178 http.status_code=403 http.response.header.nel="{\"success_fraction\":0,\"report_to\":\"cf-nel\",\"max_age\":604800}" http.response.header.alt_svc="h3=\":443\"; ma=86400" http.response_content_length=219 http.response.header.cf_ray=90c0698bee2d124e-ORD http.response.header.report_to="{\"endpoints\":[{\"url\":\"https:\/\/a.nel.cloudflare.com\/report\/v4?s=bkYoWKv7FpD9Lr8Vj8bmE8KPocsIWMjJegF5%2FeUrO8Tn5GsnSAba%2B4w9sGeZcVC4n85JSWq0EHbRCydcGWdct9N7bGDqLQ8qnQkjmhEEFDWCiCnWyYi7cPmgyOe%2Bz0BqJIxpVKiEBhSPsbWj1k%2FRtw%3D%3D\"}],\"group\":\"cf-nel\",\"max_age\":604800}" http.response.header.date="Mon, 03 Feb 2025 06:37:25 GMT"
http.response.body=
| <?xml version="1.0" encoding="UTF-8"?><Error><Code>SignatureDoesNotMatch</Code><Message></Message><RequestId>tx000004599d10cd4712119-0067a06425-1192799-default</RequestId><HostId>1192799-default-default</HostId></Error>
http.response.header.content_type=application/xml http.response.header.accept_ranges=bytes http.response.header.x_envoy_upstream_service_time=34 http.response.header.server_timing="cfL4;desc=\"?proto=TCP&rtt=25278&min_rtt=17126&rtt_var=8980&sent=8&recv=12&lost=0&retrans=0&sent_bytes=4263&recv_bytes=2255&delivery_rate=169645&cwnd=254&unsent_bytes=0&cid=6d94aa0b9071c6de&ts=116&x=0\""
2025-02-03T00:37:25.736-0600 [DEBUG] backend-s3: request failed with unretryable error https response error StatusCode: 403, RequestID: tx000004599d10cd4712119-0067a06425-1192799-default, HostID: 1192799-default-default, api error SignatureDoesNotMatch: UnknownError: aws.region=default aws.s3.bucket=<redacted-bucket-name> rpc.method=ListObjectsV2 rpc.service=S3 rpc.system=aws-api tf_aws.sdk=aws-sdk-go-v2 tf_backend.operation=Workspaces tf_backend.req_id=05ad803a-067d-c682-41aa-fab081edf53a tf_backend.s3.bucket=<redacted-bucket-name> tf_backend.workspace-prefix=env:/
╷
│ Error: Failed to get existing workspaces: Unable to list objects in S3 bucket "<redacted-bucket-name>" with prefix "env:/": operation error S3: ListObjectsV2, https response error StatusCode: 403, RequestID: tx000004599d10cd4712119-0067a06425-1192799-default, HostID: 1192799-default-default, api error SignatureDoesNotMatch: UnknownError
│
│
Expected Behavior
Should be able to use Ceph S3 Bucket for Backend State management
Actual Behavior
I am able to use AWS CLI and other AWS SDKs - like boto3 and dart's aws_signature_v4 without any issues. But when I use Terraform backend it throws the error. I don't have the issue when I am using the IP address based endpoint.
Steps to Reproduce
terraform init -migrate-state
Additional Context
It's been working great. I have 2 RGW gateways exposed to internet using Cloudflare Tunnels. I am able to use AWS CLI and SDKs (both dart and python) to interact with it using a Cloudflare DNS.. the setup is simple. I have a cloudflare tunnel running on a Kubernetes cluster (separate from the nodes actually running the RGW Gateways) -> Offloads to a Envoy Proxy -> that load balances between my RGW Gateway instances.
cat ~/.aws/config
[profile Terraform]
endpoint_url = https://<redacted-record>.arkobasu.space
region = default
output = json
alpha@Arkos-MacBook-Pro workspace
I can confirm that Accounts and IAM API are also functional. So this profile for example is created under a new Account, by the root user of the account -> that then created this user.
I have been stuck on this for a bit. I have tried everything. The Signature validation does work when I am using something like Presign for objects and accessing them over the browser.
I would appreciate it very much if you could give me some direction.
References
No response
Generative AI / LLM assisted development?
No response
Thanks for this report. S3 backend issues are handled by the HashiCorp AWS provider team. The S3 backend supports the AWS S3 service, compatibility with other S3 implementations is not necessarily supported although issues interacting with these services are frequently addressed.
You may also wish to use the community forum where there are more people ready to help. The GitHub issues here are monitored only by a few core maintainers. Thanks!
I've also find the same issue with CloudFlare Tunnel but I'm using MinIO instead of Ceph. I've debug deeper a little bit and I think the culprit is the field SignedHeaders on the Authorization HTTP header.
Basically when Terrform create a request to S3 backend, the HTTP header has SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date in the Authorization header. The request is sent with Accept-Encoding: identity. However, Cloudflare Tunnel will change this to gzip (save some traffic I assume). Because this field is changed midway through, the signature sent from Terraform is no longer valid.
In contrast, AWS-CLI client sends the same request (ListObjectsV2) with the SignedHeaders=host;x-amz-content-sha256;x-amz-date. Since Cloudflare does not touch either of these fields, the signature remains valid although Accept-Encoding is changed by the proxy.
At this point I'm not 100% how the signature and the SignedHeaders are created, whether in Terraform itself or by AWS-SDK used by Terraform.
Here's the Terraform's log:-
2025-11-25T04:28:57.057+0700 [DEBUG] backend-s3: HTTP Request Sent: aws.region=main aws.s3.bucket=boe rpc.method=ListObjectsV2 rpc.service=S3 rpc.system=aws-api tf_aws.custom_endpoint=true tf_aws.sdk=aws-sdk-go-v2 tf_backend.operation=Workspaces tf_backend.req_id=cd4fb785-d05e-fba8-d6b6-1a76f0d37b17 tf_backend.s3.bucket=boe tf_backend.workspace-prefix=env:/ http.request.header.x_amz_date=20251124T212857Z http.request.body="" http.method=GET http.url="http://s3.sleepyhead.name/boe?list-type=2&max-keys=1000&prefix=env%3A%2F" net.peer.name=s3.sleepyhead.name http.user_agent="APN/1.0 HashiCorp/1.0 Terraform/1.14.0 (+https://www.terraform.io) aws-sdk-go-v2/1.39.2 ua/2.1 os/windows lang/go#1.25.4 md/GOOS#windows md/GOARCH#amd64 api/s3#1.88.3 m/C,e" http.request.header.x_amz_content_sha256=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 http.request.header.amz_sdk_invocation_id=63552b26-8002-49e1-bbc3-63279f8ce9c3 http.request.header.accept_encoding=identity http.request.header.amz_sdk_request="attempt=1; max=5" http.request.header.authorization="AWS4-HMAC-SHA256 Credential=BBu231PFVI0a3pa6Ic0m/20251124/main/s3/aws4_request, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date, Signature=*****"
2025-11-25T04:28:57.212+0700 [DEBUG] backend-s3: HTTP Response Received: aws.region=main aws.s3.bucket=boe rpc.method=ListObjectsV2 rpc.service=S3 rpc.system=aws-api tf_aws.custom_endpoint=true tf_aws.sdk=aws-sdk-go-v2 tf_backend.operation=Workspaces tf_backend.req_id=cd4fb785-d05e-fba8-d6b6-1a76f0d37b17 tf_backend.s3.bucket=boe tf_backend.workspace-prefix=env:/ http.response.header.content_type=application/xml http.response.header.cf_cache_status=DYNAMIC http.response.header.connection=keep-alive http.response.header.accept_ranges=bytes http.response.header.x_amz_bucket_region=main http.response.header.x_ratelimit_remaining=2826
http.response.body=
| <?xml version="1.0" encoding="UTF-8"?>
| <Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message><BucketName>boe</BucketName><Resource>/boe</Resource><Region>main</Region><RequestId>187B0E9AE47B7389</RequestId><HostId>dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8</HostId></Error>
http.duration=155 http.status_code=403 http.response.header.cf_ray=9a3bffbee8c3ef6a-SIN http.response.header.date="Mon, 24 Nov 2025 21:28:57 GMT" http.response.header.alt_svc="h3=\":443\"; ma=86400" http.response.header.x_amz_request_id=187B0E9AE47B7389 http.response.header.strict_transport_security="max-age=31536000; includeSubDomains" http.response.header.x_xss_protection="1; mode=block" http.response_content_length=414 http.response.header.nel="{\"report_to\":\"cf-nel\",\"success_fraction\":0.0,\"max_age\":604800}" http.response.header.x_amz_id_2=dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8 http.response.header.vary=["Origin", "Accept-Encoding"] http.response.header.report_to="{\"group\":\"cf-nel\",\"max_age\":604800,\"endpoints\":[{\"url\":\"https://a.nel.cloudflare.com/report/v4?s=uCY%2FK0xz76S%2BPSNSCmRRT5VIAmSN6hhVGP9vqzP4TcDejja1S8zQv1gzcvhTFNNW%2F30ZAduaz2YPVBQt3TCkfFRb0fAoWAqqC8PIV3WTbG2c6OdBteL2k9b6um13zQ%3D%3D\"}]}" http.response.header.server_timing=["cfCacheStatus;desc=\"DYNAMIC\"", "cfEdge;dur=18,cfOrigin;dur=43"] http.response.header.server=cloudflare http.response.header.x_ratelimit_limit=2826 http.response.header.x_content_type_options=nosniff
Here's MinIO's log (notice the Accept-Encoding is now gzip):
s3.sleepyhead.name [REQUEST s3.ListObjectsV2] [2025-11-25T04:10:03.705] [Client
IP: [2405:9800:b651:b9b6:251e:fb4b:d833:1991]]
s3.sleepyhead.name GET /boe?list-type=2&max-keys=1000&prefix=env%3A%2F
s3.sleepyhead.name Proto: HTTP/1.1
s3.sleepyhead.name Host: s3.sleepyhead.name
s3.sleepyhead.name User-Agent: APN/1.0 HashiCorp/1.0 Terraform/1.14.0 (+https://www.terraform.io) aws-sdk-go-v2/1.39.2 ua/2.1 os/windows lang/go#1.25.4 md/GOOS#windows md/GOARCH#amd64 api/s3#1.88.3 m/C,e
s3.sleepyhead.name X-Forwarded-For: 2405:9800:b651:b9b6:251e:fb4b:d833:1991
s3.sleepyhead.name Accept-Encoding: gzip
s3.sleepyhead.name Cf-Visitor: {"scheme":"http"}
s3.sleepyhead.name Cf-Warp-Tag-Id: e71ceb25-392e-4980-b05c-aecbc85b4bb2
s3.sleepyhead.name X-Forwarded-Proto: http
s3.sleepyhead.name Amz-Sdk-Invocation-Id: 2a66b382-bf8b-4187-ba36-1d2cf94d0e43
s3.sleepyhead.name Amz-Sdk-Request: attempt=1; max=5
s3.sleepyhead.name Authorization: AWS4-HMAC-SHA256 Credential=BBu231PFVI0a3pa6Ic0m/20251124/main/s3/aws4_request, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date, Signature=6799dee63151e31b6740d923925e62e15adfc48ff5ef3fc12cd54fc1d29fa505
s3.sleepyhead.name Cf-Connecting-Ip: 2405:9800:b651:b9b6:251e:fb4b:d833:1991
s3.sleepyhead.name Cf-Ipcountry: TH
s3.sleepyhead.name X-Amz-Date: 20251124T211003Z
s3.sleepyhead.name Cdn-Loop: cloudflare; loops=1
s3.sleepyhead.name Cf-Ray: 9a3be4111ea07b48-BKK
s3.sleepyhead.name Content-Length: 0
s3.sleepyhead.name X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
s3.sleepyhead.name X-Forwarded-Host: s3.sleepyhead.name
s3.sleepyhead.name Connection: keep-alive
s3.sleepyhead.name
s3.sleepyhead.name [RESPONSE] [2025-11-25T04:10:03.705] [ Duration 243µs TTFB 230.49µs ↑ 265 B ↓ 414 B ]
s3.sleepyhead.name 403 Forbidden
s3.sleepyhead.name X-Ratelimit-Limit: 2826
s3.sleepyhead.name X-Ratelimit-Remaining: 2826
s3.sleepyhead.name X-Xss-Protection: 1; mode=block
s3.sleepyhead.name Accept-Ranges: bytes
s3.sleepyhead.name Content-Length: 414
s3.sleepyhead.name Content-Type: application/xml
s3.sleepyhead.name Strict-Transport-Security: max-age=31536000; includeSubDomains
s3.sleepyhead.name Vary: Origin,Accept-Encoding
s3.sleepyhead.name X-Amz-Bucket-Region: main
s3.sleepyhead.name X-Amz-Id-2: dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8
s3.sleepyhead.name X-Amz-Request-Id: 187B0D92EB054F63
s3.sleepyhead.name Server: MinIO
s3.sleepyhead.name X-Content-Type-Options: nosniff
s3.sleepyhead.name <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message><BucketName>boe</BucketName><Resource>/boe</Resource><Region>main</Region><RequestId>187B0D92EB054F63</RequestId><HostId>dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8</HostId></Error>
s3.sleepyhead.name
And this is MinIO's log when I uses AWS-CLI (notice that the SignedHeaders does not have content-type).
s3.sleepyhead.name [REQUEST s3.ListObjectsV2] [2025-11-25T05:10:20.587] [Client IP: [2405:9800:b651:b9b6:251e:fb4b:d833:1991]]
s3.sleepyhead.name GET /boe?list-type=2&prefix=&delimiter=%2F&encoding-type=url
s3.sleepyhead.name Proto: HTTP/1.1
s3.sleepyhead.name Host: s3.sleepyhead.name
s3.sleepyhead.name User-Agent: aws-cli/2.32.3 md/awscrt#0.28.4 ua/2.1 os/windows#11 md/arch#amd64 lang/python#3.13.9 md/pyimpl#CPython m/Z,E,N,n,C,b cfg/retry-mode#standard md/installer#exe
md/prompt#off md/command#s3.ls
s3.sleepyhead.name Authorization: AWS4-HMAC-SHA256 Credential=BBu231PFVI0a3pa6Ic0m/20251124/main/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=436e0f9d12756cc2c445691011a3f4c058e3d12c9c8abfef6bed44e6044d4b33
s3.sleepyhead.name Cf-Ray: 9a3c3c5e8831797a-SIN
s3.sleepyhead.name Accept-Encoding: gzip
s3.sleepyhead.name Cdn-Loop: cloudflare; loops=1
s3.sleepyhead.name Cf-Visitor: {"scheme":"http"}
s3.sleepyhead.name Content-Length: 0
s3.sleepyhead.name X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
s3.sleepyhead.name X-Amz-Date: 20251124T221020Z
s3.sleepyhead.name X-Forwarded-For: 2405:9800:b651:b9b6:251e:fb4b:d833:1991
s3.sleepyhead.name Cf-Connecting-Ip: 2405:9800:b651:b9b6:251e:fb4b:d833:1991
s3.sleepyhead.name Cf-Ipcountry: TH
s3.sleepyhead.name X-Forwarded-Proto: http
s3.sleepyhead.name Cf-Warp-Tag-Id: e71ceb25-392e-4980-b05c-aecbc85b4bb2
s3.sleepyhead.name Connection: keep-alive
s3.sleepyhead.name
s3.sleepyhead.name [RESPONSE] [2025-11-25T05:10:20.589] [ Duration 1.067ms TTFB 1.039108ms ↑ 210 B ↓ 344 B ]
s3.sleepyhead.name 200 OK
s3.sleepyhead.name Server: MinIO
s3.sleepyhead.name Strict-Transport-Security: max-age=31536000; includeSubDomains
s3.sleepyhead.name Vary: Origin,Accept-Encoding
s3.sleepyhead.name X-Amz-Request-Id: 187B10DD0A0A0894
s3.sleepyhead.name X-Content-Type-Options: nosniff
s3.sleepyhead.name X-Xss-Protection: 1; mode=block
s3.sleepyhead.name Accept-Ranges: bytes
s3.sleepyhead.name X-Amz-Bucket-Region: main
s3.sleepyhead.name X-Amz-Id-2: dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8
s3.sleepyhead.name X-Ratelimit-Limit: 2826
s3.sleepyhead.name X-Ratelimit-Remaining: 2826
s3.sleepyhead.name Content-Length: 344
s3.sleepyhead.name Content-Type: application/xml
s3.sleepyhead.name <?xml version="1.0" encoding="UTF-8"?>
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>boe</Name><Prefix></Prefix><KeyCount>1</KeyCount><MaxKeys>1000</MaxKeys><Delimiter>/</Delimiter><IsTruncated>false</IsTruncated><CommonPrefixes><Prefix>dbi/</Prefix></CommonPrefixes><EncodingType>url</EncodingType></ListBucketResult>
s3.sleepyhead.name
It's the aws go sdk from what I have realized. If you use version 1.5.7 and older it will work. Since it uses an older version of the sdk where the issue doesn't happen.