Disambiguate backend configurations during migration

[remilapeyre]

A user reported at https://github.com/hashicorp/terraform-provider-consul/issues/275 that the message when migrating backend was not detailed enough to be sure what was the previously configured backend, and what is the new one. This can make migrating from one backend to another dangerous as there is no way to check the configuration is correct when actually migrating to a new backend.

The message when using the consul backend (though the issue still stands with other backends as well) was:

Do you want to migrate all workspaces to "consul"?
  Both the existing "consul" backend and the newly configured "consul" backend
  support workspaces. When migrating between backends, Terraform will copy
  all workspaces (with the same names). THIS WILL OVERWRITE any conflicting
  states in the destination.

  Terraform initialization doesn't currently migrate only select workspaces.
  If you want to migrate a select number of workspaces, you must manually
  pull and push those states.

  If you answer "yes", Terraform will migrate all states. If you answer
  "no", Terraform will abort.

I changed the Backend interface to add a String() method that returns an HCL representation of the backend and used it to return more information when migrating. The new message is:

Do you want to copy existing state to the new backend?
  Pre-existing state was found while migrating the previous "local" backend to the
  newly configured "consul" backend. No existing state was found in the newly
  configured "consul" backend.

  The configuration for the previous "local" backend was:

    backend "local" {
      path          = ""
      workspace_dir = ""
    }

  The configuration for the new "consul" backend is:

    backend "consul" {
      access_token = ""
      address      = ""
      ca_file      = ""
      cert_file    = ""
      datacenter   = ""
      gzip         = false
      http_auth    = ""
      key_file     = ""
      lock         = true
      path         = "full/path"
      scheme       = ""
    }

  Do you want to copy this state to the new "consul"
  backend? Enter "yes" to copy and "no" to start with an empty state.

For the local and remote backends the implementation is straightforward, for all the others the representation is generated from the backend's Schema and its configuration.

[codecov-commenter]

Codecov Report

Merging #29574 (b1271dd) into main (21228e1) will increase coverage by 0.83%. The diff coverage is 32.78%.

:exclamation: Current head b1271dd differs from pull request most recent head c7e8b82. Consider uploading reports for the commit c7e8b82 to get more accurate results

:mega: This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more

Additional details and impacted files

Impacted Files	Coverage Δ
backend/backend.go	45.83% <ø> (+14.58%)	:arrow_up:
backend/local/backend.go	48.51% <0.00%> (+1.81%)	:arrow_up:
backend/remote-state/cos/backend.go	0.00% <0.00%> (ø)
backend/remote-state/etcdv2/backend.go	0.00% <0.00%> (ø)
backend/remote-state/etcdv3/backend.go	0.00% <0.00%> (ø)
backend/remote-state/gcs/backend.go	0.00% <0.00%> (ø)
backend/remote-state/kubernetes/backend.go	0.00% <0.00%> (ø)
backend/remote-state/manta/backend.go	0.00% <0.00%> (ø)
backend/remote-state/oss/backend.go	0.00% <0.00%> (ø)
backend/remote-state/pg/backend.go	0.00% <0.00%> (ø)
... and 595 more

[hashicorp-cla]

All committers have signed the CLA.

[crw]

Hi @remilapeyre, thanks for this submission! I assume this is still a valid and reasonable thing to want from backend migration output. I will raise it with the team to see if this can be reviewed. Thanks again!

[remilapeyre]

Hi @remilapeyre, thanks for this submission! I assume this is still a valid and reasonable thing to want from backend migration output. I will raise it with the team to see if this can be reviewed. Thanks again!

Hi @crw, do you know if this will be reviewed?

[crw]

Hi @remilapeyre, I will raise it for 1.4. The initial feedback was positive.

[remilapeyre]

Hi @crw, I fixed the conflicts, the code should be ready for review.

[crw]

Thanks @remilapeyre. This is on our radar for review.

[apparentlymart]

Hi @remilapeyre! Thanks for working on this.

One small concern I have about this approach is that so far we've tried to draw a line where "plugins" don't ever directly generate HCL syntax but instead generate data structures that Terraform CLI/Core can use to generate HCL. This means that if we evolve the language in future we should be able to tweak the details of what exactly these things generate without having to change every plugin.

Backends are of course not currently actually plugins, but architecturally we treat them as such to minimize situations where we'd need to go and mechanically make the same change in every backend. We are also trying to keep the architecture such that we could potentially make at least the state-storage-only backends plugins in future, although that isn't quite possible yet due to some details of how the backend abstraction works today.

With all of that said then: do you think it would be plausible for the backend to return its configuration as a cty.Value of object type, and then have the UI layer turn that into something resembling HCL native syntax by reference to the backend's configuration schema? That would then make this mechanism similar to how terraform plan presents resource instance attributes and blocks using an HCL-like syntax, where the exact details of what's being generated are subject to change at the UI in future.

I must admit I've not checked yet how reusable the new terraform plan diff rendering code would be for rendering an arbitrary cty.Value returned from a backend -- hopefully at least parts of that are general enough to be borrowed here -- but the way the plan diffs (and similar mechanisms like terraform show) are implemented at least illustrates that an architecture like this should be possible in principle, and would avoid the need for backends to depend directly on anything from HCL.

Thanks again!