terraform
terraform copied to clipboard
hashicorp
"Unexpected end of JSON input" error locking s3 backend
Terraform Version
Terraform v1.3.8
on linux_amd64
Terraform Configuration Files
terraform {
backend "s3" {
bucket = "<redacted>"
key = "bug/unexpected-end-of-json-input/terraform.tfstate"
encrypt = true
kms_key_id = "<redacted>"
dynamodb_table = "<redacted>"
}
}
Debug Output
https://gist.github.com/petur/b66b6c7fce877ddb1bf3cc7629101de0
Expected Behavior
When terraform is run with -lock-timeout, it should either acquire the lock, or retry until the timeout expires.
Actual Behavior
Terraform fails with the error:
╷
│ Error: Error acquiring the state lock
│
│ Error message: 2 errors occurred:
│ * ConditionalCheckFailedException: The conditional request failed
│ * unexpected end of JSON input
│
│
│
│ Terraform acquires a state lock to protect the state from being written
│ by multiple users at the same time. Please resolve the issue above and try
│ again. For most commands, you can disable locking with the "-lock=false"
│ flag, but this is not recommended.
╵
Steps to Reproduce
-
terraform init - Duplicate the folder containing the configuration
- Run
while TF_LOG=trace terraform plan -lock-timeout=60s; do : ; donein both folders (needs two shells).
(This is an artificial way to reproduce the issue. It was found in CI where multiple people push changes and plans get run in parallel.)
Additional Context
No response
References
No response
This looks like a simple race condition in the locking code:
In s3/client.go:
_, err := c.dynClient.PutItem(putParams)
if err != nil {
lockInfo, infoErr := c.getLockInfo()
if infoErr != nil {
err = multierror.Append(err, infoErr)
}
lockErr := &statemgr.LockError{
Err: err,
Info: lockInfo,
}
return "", lockErr
}
PutItem fails with a ConditionalCheckFailedException when another user has the state locked. The other user then unlocks the state before c.getLockInfo manages to fetch the lock.
In locker.go:
if le == nil || le.Info == nil || le.Info.ID == "" {
// If we don't have a complete LockError then there's something
// wrong with the lock.
return "", err
}
Here le.Info.ID is presumably empty because the lock ID was gone before getLockInfo could read it from DynamoDB, and the locker exits the wait loop.
╷ │ Error: Error acquiring the state lock │ │ Error message: 2 errors occurred: │ * ConditionalCheckFailedException: The conditional request failed │ * unexpected end of JSON input │ │ │ │ Terraform acquires a state lock to protect the state from being written │ by multiple users at the same time. Please resolve the issue above and try │ again. For most commands, you can disable locking with the "-lock=false" │ flag, but this is not recommended.
