terraform
terraform copied to clipboard
hashicorp
Terraform console creates a lock that does not go away before it's closed
Terraform Version
Terraform v0.12.14
Expected Behavior
When i run "terraform console" I expect it to to only lock state remotely long enough to do data lookups. Once the prompt is displayed, I would expect the lock to be released.
Actual Behavior
When i run "terraform console" it creates a lock in the remote state that isn't released until I close the console.
Steps to Reproduce
- run
terraform consoleon a project with shared state on machine A, leave the console open - run
terraform consoleon a project with shared state on machine B - bow head in sadness
Could I check how this issue proceed?
I have received similar issues from my customer when run terraform console
tfstate.lock.info
{"ID":"753dd145-4007-3fb5-00c9-58876ea44d8a","Operation":"OperationTypeInvalid","Info":"","Who":"WIN-5P0ODUDEVVF\Administrator@LGUPLUS-PC","Version":"0.14.2","Created":"2020-12-18T06:36:20.2570009Z","Path":"terraform.tfstate"}
$ terraform plan
Error: Error locking state: Error acquiring the state lock: Failed to read state file: The state file could not be read: read terraform.tfstate: The process cannot access the file because another process has locked a portion of the file.
Terraform acquires a state lock to protect the state from being written by multiple users at the same time. Please resolve the issue above and try again. For most commands, you can disable locking with the "-lock=false" flag, but this is not recommended.
And then terraform force-unlock not working
$ terraform force-unlock 753dd145-4007-3fb5-00c9-58876ea44d8a Local state cannot be unlocked by another process
Best would be to add an argument --no-state-lock to the terraform console command.
@tringuyen-yw better still to implement it as -lock=false just like terraform plan has
Any update on this, it's been an open issue for a long time. When you strike the issue the error says to run with -lock=false. but this does not work so this makes it a bug.
This would be a welcome feature. Our terraform developers dont have write access to the gcp bucket where we keep our tfstate. They run plan with -lock=false to see changes which will be applied by a CICD after a PR is merged. These developers will not be able to run terraform console for debugging because there is no -lock flag for terraform console
Hi all. May I implement this improvement? I’m interested in terraform contribution and this issue is a good first one.
Hi @kazuki-hanai, before you work on a pull request, please check out https://github.com/hashicorp/terraform/blob/main/.github/CONTRIBUTING.md#proposing-a-change - specifically:
In order to be respectful of the time of community contributors, we aim to discuss potential changes in GitHub issues prior to implementation. That will allow us to give design feedback up front and set expectations about the scope of the change, and, for larger changes, how best to approach the work such that the Terraform team can review it and merge it along with other concurrent work.
This would be the first step to making this contribution, so if you do decide to move forward please start with the discussion of a potential solution.
I raised this issue in a triage meeting in May 2022. That meeting did not have the benefit of the later use cases. Here is an overview of the feedback:
Broadly speaking, the console feature is intended for configuration language experimentation, rather than to be built into production workflows (with all respect to anyone who is using console in such a workflow). We want console to be as safe to use as possible. Another core assumption is that the console will not be left open indefinitely, unreasonably holding open locks.
One possible solution would be to acquire and release a lock on every statement execution. This solution would have a relatively high overhead, but is more feasible than a command-line option that would disable locks entirely.
In general, the team is not adding new command line options. So a solution that avoided adding a new argument would be preferred.
I hope this helps guide you in the right direction when proposing a solution!
I don't know why it should keep a lock. Last time I checked, it couldn't modify anything (may have changed). What I, and my team mates used it for, was debugging the contents of calculated variables.
Hi @crw, I don't get your argumentation.
Following case:
- Terraform was working for me in a toy project, workspace on NFS.
- Because of changes in the NFS setup, it started to fail recently with "Error: Error acquiring the state lock" in commands which need a lock.
Where I don't doubt that it is a good thing to hold a lock, I'm a bit lost here:
- terraform [plan|apply] -lock=false -- work as expected, even if potentially dangerous (not in my use case)
- terraform console -- does not work, fails with "Error: Error acquiring the state lock"
- terraform console -lock=false -- does not work, fails because it does not implement -lock=false
Now you argue to have "console as save to use as possible" and thus not want to implement -lock=false leaves the thing inconsistent. The real dangerous "apply" allows -lock=false. The usually non-dangerous "console" does not, removing the claimed used as "language experimentation" prior to "apply".
Thanks for the great work :+1:
I'm also confused as to why the console would need to create a lock file if it can't actually change anything. From my understanding, it only reads the current state file from the defined backend and can not make changes to the current state. Correct me if I'm wrong here.
This debugging tool is super useful, and should never be used in a production pipeline. But when we are debugging with this tool and our pipeline runs while a developer is using the tool for debugging reasons, the lock file is present in remote storage and the pipeline fails. This should not be the intended function of a debugging tool.
Apologies for the late response. The team would be happy to consider a PR that adds a -lock=false to terraform console. I was informed that you should be able to look at the implementation from the other sections of code where this exists for guidance on how to implement it. Thanks!
