Inconsistent state on lost/deleted backend situation

[johnmarcou]

Hello,

I found an issue during my tests. It's a bit strange test I recognise, but I think it could be interesting to raise this issue since the situation may happen.

If you terraform apply to create a DNS record on CoreDNS, then you manually delete the record on CoreDNS (or you lost your CoreDNS backend), the next terraform apply should check/re-create the resource, but this crash the terraform command.

In a real-world context, let's say: "the network team decides to upgrade their DNS infrastructure, and lose all the data". In this situation, you want to be able to re-run terraform apply to repopulate the DNS zones.

How to reproduce

Define a simple DNS record:

provider "coredns" {
    etcd_endpoints = "http://127.0.0.1:2379"
    zones = "skydns.local"
}

resource "coredns_record" "foo" {
    fqdn = "foo.skydns.local"
    type = "A"
    rdata = [ "10.10.10.10", "10.10.10.20" ]
    ttl = "60"
}

Create it:

# terraform apply
...
Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

Now, delete/lose your CoreDNS server etcd backend, or just delete the record manualy. In this testing case, I just restart testing coredns and etcd docker containers, which are running without any external volumes, so no persistent data here.

docker-compose down; docker-compose up

Then apply it again:

# terraform apply
coredns_record.foo: Refreshing state... (ID: foo.skydns.local.)
Error: Error refreshing state: 1 error(s) occurred:
* coredns_record.foo: 1 error(s) occurred:
* coredns_record.foo: coredns_record.foo: Failed to get DNS records

Comments

I tried this situation with another providers, like VMware vSphere. I deployed a VM with Terraform, then I deleted it manually from the vSphere interface. A second terraform apply will re-create the VM to match the current state with the desired state.

My understanding is, in the situation where the resource has been "lost/deleted", terraform apply read the tfstate and desired state, check if the actual resource still exist, if not, it decided to recreate it, considering the tfstate is not consistent anymore.

Nota: Deleting terraform.tfstate is not a good workaround, since it will re-create all the defined resources, not only the coredns ones.

I feel like it could be related with the previous issue, may be something about "resource identification/tracking"?

Thank you a lot!