(maybe) 1.12 S3 Backend (nonAWS) Regression: skip_s3_checksum ignored?

[gr-fl]

Terraform Version

1.12.1

Terraform Configuration Files

# Configure TF remote-state
terraform {
  backend "s3" {
    bucket = "tf-state-terraform-myorg"
    key    = "org/prod/project/.terraform.tfstate"
    region = "somewhere" # Required, but in our case ignored
    #access_key = ENV_AWS_ACCESS_KEY_ID # set in env, not via tf
    #secret_key = ENV_AWS_SECRET_ACCESS_KEY # set in env, not via tf
    skip_credentials_validation = true
    skip_region_validation      = true
    skip_requesting_account_id  = true
    skip_metadata_api_check     = true
    skip_s3_checksum            = true
    endpoints = {
      s3 = "https://internal-s3.prod.m/" # our custom internal S3
    }
    use_path_style = true
  }
}

Today we migrated a project from using a consul backend to an s3 backend.

After running terraform-init -migrate-state, I immediately see issues with checksums showing up in the state even though the option to skip checksums is enabled.

$ terraform init
Initializing the backend...
╷
│ Error: Error refreshing state: 2 problems:
│
│ - Unsupported state file format: The state file could not be parsed as JSON: syntax error at byte offset 9.
│ - Unsupported state file format: The state file does not have a "version" attribute, which is required to identify the format version.

Sure enough, if I manually pull the state file, I can see things like this in the middle of my nice JSON state:

5000
{
    "version": 4,
...
          },
     
0
x-amz-checksum-crc32:jcuQew==

1d37ef
     "sensitive_attributes": [],
   ...
} 

I know this was an issue in earlier TF versions, but we successfully set up a project in terraform 1.8 using the S3 provider. This is the first project on TF 1.12 to get moved to S3 from consul.

I considered moving the 1.12 project to 1.8, but that seems like a lot of work, and I'm more posting this out of curiosity to see if anyone else has had this issue come up.

Also, I even tried fixing the state manually after migration (cleaning out checksum stuff) and uploading a cleaned state copy. This gets the project in a state to plan and apply successfully, but after apply, the state will be broken again.

Debug Output

...debug output, or link to a gist...

Expected Behavior

State file should have been created in S3 without any checksums.

Actual Behavior

Checksums are present in the statefile uploaded to S3 bucket.

Steps to Reproduce

1. Add config for S3 backend
2. Run terraform init -migrate-state
3. Run terraform init

Additional Context

This is not utilizing AWS' S3 implementation, rather a S3 variant solution, but the same bucket works with terraform 1.8.

References

• https://developer.hashicorp.com/terraform/language/backend/s3#skip_s3_checksum
• Maybe? https://github.com/hashicorp/terraform/issues/34053

Generative AI / LLM assisted development?

N/A

[crw]

Thanks for this report, I will pass it on to the AWS Provider team at HashiCorp, who maintain this backend.

[gr-fl]

I discovered something a little bit interesting. I turned off the skip_s3_checksum flag to see what would happen, expecting a similar result. Actually, my JSON still got litered with checksums. But they are different now.

Original - if skip_s3_checksum is TRUE:

500000
{
  "version": 4,
  "terraform_version": "1.12.1",
...
              "somejsonvalue",
   
0
x-amz-checksum-crc32:K1edIQ==

1d2894
           "someotherjsonvalue",
           "somemorejsonforyou",
...
}
0
x-amz-checksum-crc32:Oy5svw==

If skip_s3_checksum is FALSE:

500000
{
  "version": 4,
  "terraform_version": "1.12.1",
...
              "somejsonvalue",
   
0
x-amz-checksum-sha256:y48GdabcD12cquHd05RESOvf/Ywqkzau3wycO8H4HmM=

1d2894
           "someotherjsonvalue",
           "somemorejsonforyou",
...
}
0
x-amz-checksum-sha256:NjWGNQmpbBXNkd9ELQQQuooXPocjaZWcbkKtGkrCo+g=

To me personally this does point to something odd with the underlying package. It does look like there was an update to this dependency: https://github.com/hashicorp/terraform/commit/8be88f76ea0b9a27a1a934f6340f70afbcce7a87#diff-1477bba9710c7ea1c566da2da37d964a0eb7f020ffa91352cb8493cade3e0611

Should I chase elsewhere? or stay on this issue?

[gr-fl]

Hi, I couldn't leave well enough alone, so I downgraded our project to 1.8.

The state is now being written correctly to S3 without checksums. This was the process I used to try to downgrade our project:

1. wget https://releases.hashicorp.com/terraform/1.8.2/terraform_1.8.2_linux_amd64.zip
2. unzip terraform_1.8.2_linux_amd64.zip
3. mv terraform ~/bin/terraform-1.8.2  # (in path)
4. terraform-1.8.2 version # should produce right thing
6. terraform-1.8.2 -reconfigure
7. terraform-1.8.2 -migrate-state
9. terraform-1.8.2 plan -o 1.tf.plan
11. terraform-1.8.2 apply 1.tf.plan
12. terraform-1.8.2 init # works this time, state file in bucket was written without checksums, so can check state successfully

edit: a prerequisite is modifying the version file to point to version 1.8.2, then adding a small change that will show up in the plan/apply. This is fairly obvious you'd think but I felt maybe I should add it in case it comes up. To me, this would indicate a difference in checksum behavior due to terraform version changing. I know 4 subversions apart is a fairly big diff to look through though. Let me know how else I can help.

[claytonpeters]

I took a quick look at this and I think this is likely due to a change in the aws-sdk-go-v2 library. From the release notes on 2025-01-15 for that package it says that the default checksum algorithm is CRC32. I've found an announcement about this change at https://github.com/aws/aws-sdk-go-v2/discussions/2960

In the S3 backend code in Terraform, it only sets the algorithm when skipS3Checksum in RemoteClient is false, and leaves it unset when true, which I'm guessing is now causing the aws-sdk-go-v2 library to use it's default of CRC32. This would also explain the difference in checksums that @gr-fl is seeing, where CRC32 is being used when skipS3Checksum is set to false.

I think this will require RequestChecksumCalculation in the Options object passed to s3.NewFromConfig will need setting to aws.RequestChecksumCalculationWhenRequired (as per the last comment on that announcement)

[niuzhenguo]

I'm having the same issue, any update or workaround on this?

[nicowarss]

As workaround, i downgraded terraform version from 1.12 to 1.10 and is working well

[gpietrus-akamai]

Try these env variables

  AWS_RESPONSE_CHECKSUM_VALIDATION: when_required
  AWS_REQUEST_CHECKSUM_CALCULATION: when_required

[stephen-bracken]

Hello, the checksum issue is due to a breaking change in AWS involving client-side cyclic redundancy checks, agree with @gpietrus-akamai the workaround is to set checksums to when_required instead of when_supported in the aws client configuration. I encountered the same issue when upgrading to boto3 https://github.com/boto/boto3/issues/4392

https://aws.amazon.com/blogs/aws/introducing-default-data-integrity-protections-for-new-objects-in-amazon-s3/

[justinretzolk]

It looks like this is ultimately a duplicate of #36704, which also has quite a few comments from folks testing the environment variables above to positive results. With that in mind, I'm going to close this issue out so that we can keep the conversation contained in one place.

[github-actions[bot]]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.