Terraform stuck in a prefix-lists configuration change loop

[ebarrett-Ocient]

I'm noticing some very strange behavior with prefix-lists since upgrading from 1.14.0 to 1.21.0. I have the following resource defined in my terraform code:

resource "fortios_router_prefixlist" "edge_out" {
  name = "edge-out"
  rule {
    id     = "1"
    prefix = "169.254.0.0 255.255.0.0"
    le     = "32"
    action = "deny"
  }
  rule {
    id     = "100"
    prefix = "any"
  }
}

On my Fortigate, my configuration looks like the following:

config router prefix-list
    edit "edge-out"
        config rule
            edit 1
                set action deny
                set prefix 169.254.0.0 255.255.0.0
                unset ge
                set le 32
            next
            edit 100
                set prefix any
                unset ge
                unset le
            next
        end
    next
end

When I run terraform plan, it wants to make the following changes:

ebarrett lab01-fw [SDE-5095] % terraform plan -target=fortios_router_prefixlist.edge_out
data.terraform_remote_state.vault: Reading...
data.terraform_remote_state.vault: Read complete after 2s
data.vault_kv_secret_v2.fortigate_token: Reading...
data.vault_kv_secret_v2.fortigate_token: Read complete after 0s [id=kvv2/data/fortigate]
fortios_router_prefixlist.edge_out: Refreshing state... [id=edge-out]

Terraform used the selected providers to generate the following execution plan.
Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # fortios_router_prefixlist.edge_out will be updated in-place
  ~ resource "fortios_router_prefixlist" "edge_out" {
        id                    = "edge-out"
        name                  = "edge-out"
        # (2 unchanged attributes hidden)

      ~ rule {
          - flags  = 4 -> null
            id     = 1
            # (4 unchanged attributes hidden)
        }
      ~ rule {
          - flags  = 1 -> null
            id     = 100
            # (4 unchanged attributes hidden)
        }
    }

Plan: 0 to add, 1 to change, 0 to destroy.
╷
│ Warning: Resource targeting is in effect
│
│ You are creating a plan with the -target option, which means that the result
│ of this plan may not represent all of the changes requested by the current
│ configuration.
│
│ The -target option is not for routine use, and is provided only for
│ exceptional situations such as recovering from errors or mistakes, or when
│ Terraform specifically suggests to use it as part of an error message.
╵

───────────────────────────────────────────────────────────────────────────────

Note: You didn't use the -out option to save this plan, so Terraform can't
guarantee to take exactly these actions if you run "terraform apply" now.

If I run terraform apply and accept the changes, my Fortigate configuration then looks like the following:

lab01-fw01 (edge-out) # show
config router prefix-list
    edit "edge-out"
        config rule
            edit 1
                set action deny
                set prefix 169.254.0.0 255.255.0.0
                unset ge
                unset le
            next
            edit 100
                set prefix 255.255.255.255 255.255.255.255
                unset ge
                unset le
            next
        end
    next
end

As you can see, under edit 1, le is no longer set to 32 and, under edit 100, my prefix is set to 255.255.255.255 255.255.255.255 instead of any.

At this point, if I run terraform apply, it wants to revert those changes:

ebarrett lab01-fw [SDE-5095] % terraform apply -target=fortios_router_prefixlist.edge_out
data.terraform_remote_state.vault: Reading...
data.terraform_remote_state.vault: Read complete after 2s
data.vault_kv_secret_v2.fortigate_token: Reading...
data.vault_kv_secret_v2.fortigate_token: Read complete after 0s [id=kvv2/data/fortigate]
fortios_router_prefixlist.edge_out: Refreshing state... [id=edge-out]

Terraform used the selected providers to generate the following execution plan.
Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # fortios_router_prefixlist.edge_out will be updated in-place
  ~ resource "fortios_router_prefixlist" "edge_out" {
        id                    = "edge-out"
        name                  = "edge-out"
        # (2 unchanged attributes hidden)

      ~ rule {
            id     = 1
          ~ le     = 0 -> 32
            # (4 unchanged attributes hidden)
        }
      ~ rule {
            id     = 100
          ~ prefix = "255.255.255.255 255.255.255.255" -> "any"
            # (4 unchanged attributes hidden)
        }
    }

Plan: 0 to add, 1 to change, 0 to destroy.
╷
│ Warning: Resource targeting is in effect
│
│ You are creating a plan with the -target option, which means that the result
│ of this plan may not represent all of the changes requested by the current
│ configuration.
│
│ The -target option is not for routine use, and is provided only for
│ exceptional situations such as recovering from errors or mistakes, or when
│ Terraform specifically suggests to use it as part of an error message.
╵

If I accept those changes, we're back we're we started in terms of configuration on the Fortigate. However, if I run terraform plan again, it once again wants to update my flags:

ebarrett lab01-fw [SDE-5095] % terraform plan -target=fortios_router_prefixlist.edge_out
data.terraform_remote_state.vault: Reading...
data.terraform_remote_state.vault: Read complete after 2s
data.vault_kv_secret_v2.fortigate_token: Reading...
data.vault_kv_secret_v2.fortigate_token: Read complete after 0s [id=kvv2/data/fortigate]
fortios_router_prefixlist.edge_out: Refreshing state... [id=edge-out]

Terraform used the selected providers to generate the following execution plan.
Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # fortios_router_prefixlist.edge_out will be updated in-place
  ~ resource "fortios_router_prefixlist" "edge_out" {
        id                    = "edge-out"
        name                  = "edge-out"
        # (2 unchanged attributes hidden)

      ~ rule {
          - flags  = 4 -> null
            id     = 1
            # (4 unchanged attributes hidden)
        }
      ~ rule {
          - flags  = 1 -> null
            id     = 100
            # (4 unchanged attributes hidden)
        }
    }

Plan: 0 to add, 1 to change, 0 to destroy.
╷
│ Warning: Resource targeting is in effect
│
│ You are creating a plan with the -target option, which means that the result
│ of this plan may not represent all of the changes requested by the current
│ configuration.
│
│ The -target option is not for routine use, and is provided only for
│ exceptional situations such as recovering from errors or mistakes, or when
│ Terraform specifically suggests to use it as part of an error message.
╵

───────────────────────────────────────────────────────────────────────────────

Note: You didn't use the -out option to save this plan, so Terraform can't
guarantee to take exactly these actions if you run "terraform apply" now.

If I accept these changes, it once again makes edits to my le and prefix values. Essentially, terraform is stuck in a loop where it either updates those two values or reverts them back to the intended configuration over and over.

As a workaround, I have added the flags values terraform wants to apply to my configuration to my code:

resource "fortios_router_prefixlist" "edge_out" {
  name = "edge-out"
  rule {
    id     = "1"
    prefix = "169.254.0.0 255.255.0.0"
    le     = "32"
    action = "deny"
    flags  = 4
  }
  rule {
    id     = "100"
    prefix = "any"
    flags  = 1
  }
}

This is definitely more than a little hokey as the flags config evidently doesn't do anything, but it seems to have resolved my issue for now.

[ebarrett-Ocient]

I also have similar issues with the fortios_router_routemap resource. I won't get into details here, but the workaround to keep terraform from making unwanted changes was to configure the match_flags and set_flags vaules.

[MaxxLiu22]

Hi @ebarrett-Ocient

Thank you for bringing this issue to my attention. I was able to reproduce it, and it appears that the flag argument is hidden from the FOS CLI but is still functional via the API. I have reported this to the development team for resolution.

Would it be possible for you to share one of your fortios_router_routemap configurations? I believe the match_flags and set_flags might be accessible when specific arguments are configured, but I haven’t yet found the way.

Please feel free to reach out if you have any further questions.

Thanks, Maxx

[ebarrett-Ocient]

@MaxxLiu22

Sure, here's one that was giving me trouble:

resource "fortios_router_routemap" "edge_in" {
  name = "edge-in"
  rule {
    id                   = "10"
    match_community      = "GSHUT"
    set_local_preference = "1"
  }
  rule {
    id = "100"
  }
}

I found that terraform would remove my local preference config the first time I ran terraform apply with this one.

There were also a number of other resources that ended up causing issues. If I get some time later this afternoon, I can provide you with some more details.

[chriswiggins]

We are having the same issue moving from 1.20.0 to 1.21.0. Rolling back to 1.20.0 fixed the problems. the match_flags and set_flags were causing the apply to continuously keep having to update values (one apply would set some things correctly, but clearing the flags caused the others to reset).

It definitely has something to do with the null support

[MaxxLiu22]

Hi @chriswiggins ,

We sincerely apologize for the inconvenience. We have documented this issue and are currently monitoring for any similar occurrences. This issue arises from Terraform FOS provider’s new support for the unset feature when certain configurations are not specified in the TF files. If you encounter the same issue with other resources, it would greatly assist us in addressing them collectively. Once again, we apologize for any trouble this may have caused and appreciate your understanding.

Thanks, Maxx

[ebarrett-Ocient]

@MaxxLiu22 Here are a few other issues I had when upgrading to 1.21.0:

• I needed to add syslog_type = 1 to my fortios_logsyslogd_setting resource. Otherwise, terraform wants to make the following changes every time I run terraform apply:

  # fortios_logsyslogd_setting.syslog will be updated in-place
  ~ resource "fortios_logsyslogd_setting" "syslog" {
        id                      = "LogSyslogdSetting"
      - syslog_type             = 1 -> null
        # (13 unchanged attributes hidden)
    }

• Terraform repeatedly wants to make the following changes to my fortios_router_bgp configuration unless I add an id to my network6 config:

  # fortios_router_bgp.bgp_config will be updated in-place
  ~ resource "fortios_router_bgp" "bgp_config" {
        id                                 = "RouterBgp"
        # (49 unchanged attributes hidden)

      ~ network6 {
          - id                   = 1 -> null
            # (3 unchanged attributes hidden)
        }

        # (17 unchanged blocks hidden)
    }

• When running terraform apply against a fortios_router_communitylist, terraform wanted to make the following changes:

  ~ resource "fortios_router_communitylist" "gshut" {
      + get_all_tables        = "false"
        id                    = "GSHUT"
        name                  = "GSHUT"
        # (2 unchanged attributes hidden)

      ~ rule {
            id     = 1
          - regexp = "65535:0" -> null
            # (2 unchanged attributes hidden)
        }
    }

After accepting this and running terraform apply again, terraform no longer wanted to make any changes. However, no actual changes were made to my community-list configuration:

config router community-list
    edit "GSHUT"
        config rule
            edit 1
                set action permit
                set match "65535:0"
            next
        end
    next
end

• Need to add an id to each reserved_address under a fortios_systemdhcp_server resource or else terraform wants to make the following changes every time you run terraform apply:

  ~ resource "fortios_systemdhcp_server" "fortilink" {
      + get_all_tables               = "false"
        id                           = "3"
        # (40 unchanged attributes hidden)

      ~ reserved_address {
          - id              = 1 -> null
            # (6 unchanged attributes hidden)
        }
      ~ reserved_address {
          - id              = 2 -> null
            # (6 unchanged attributes hidden)
        }
      ~ reserved_address {
          - id              = 3 -> null
            # (6 unchanged attributes hidden)
        }

        # (3 unchanged blocks hidden)
    }

• Finally, I needed to add an id to the match configuration of a fortios_user_group resource or else terraform would want to remove it every time:

  # fortios_user_group.sslvpn_techops will be updated in-place
  ~ resource "fortios_user_group" "sslvpn_techops" {
        id                       = "sslvpn-techops"
        name                     = "sslvpn-techops"
        # (19 unchanged attributes hidden)

      ~ match {
          - id          = 1 -> null
            # (2 unchanged attributes hidden)
        }

        # (1 unchanged block hidden)
    }

Let me know if you'd like any more specific details about any of these.

[MaxxLiu22]

Hi @ebarrett-Ocient ,

Thank you so much for your support of our community. The information you've provided is sufficient for us to resolve the issue. I have documented all the resources you mentioned.

Thanks, Maxx

[zapotah]

The prefix lists apply loop also has an issue with CIDR notation even with 1.20.0. For example:

 # module.foo.module.common.fortios_router_prefixlist.foo-int-out will be updated in-place

  ~ resource "fortios_router_prefixlist" "foo-int-out" {
        id                    = "foo-int-out"
        name                  = "foo-int-out"
        # (3 unchanged attributes hidden)

      ~ rule {
            id     = 10
          ~ prefix = "10.10.10.0 255.255.255.0" -> "10.10.10.0/24"
            # (4 unchanged attributes hidden)
        }
      ~ rule {
            id     = 15
          ~ prefix = "192.168.0.1 255.255.255.255" -> "192.168.0.1/32"
            # (4 unchanged attributes hidden)
        }
    }

this is with a resource such as this in the common module:

resource "fortios_router_prefixlist" "foo-int-out" {
  name = "foo-int-out"
  rule {
    id = 10
    prefix = var.site_foo_object.subnet
    action = "permit"
  }
  rule {
    id = 15
    prefix = var.internal_foo_address.subnet
    action = "permit"
  }
}

where the variables are cidr string attributes from objects passed from parent module.

[ebarrett-Ocient]

Upgrading from FortiOS 7.0.14 to 7.2.10 resulted in terraform wanting to make further changes. First, it wanted to update the match_flags and set_flags in a couple of my (but not all) fortios_router_routemap resources:

  # fortios_router_routemap.edge_in will be updated in-place
  ~ resource "fortios_router_routemap" "edge_in" {
        id                    = "edge-in"
        name                  = "edge-in"
        # (2 unchanged attributes hidden)

      ~ rule {
            id                                     = 10
          ~ match_flags                            = 0 -> 16
            # (28 unchanged attributes hidden)
        }
      ~ rule {
            id                                     = 100
          ~ match_flags                            = 0 -> 16
          ~ set_flags                              = 0 -> 4
            # (26 unchanged attributes hidden)
        }
    }

Here's my terraform code for this resource:

resource "fortios_router_routemap" "edge_in" {
  name = "edge-in"
  rule {
    id                   = "10"
    match_community      = "GSHUT"
    set_local_preference = "1"
    match_flags          = 16
    set_flags            = 512
  }
  rule {
    id          = "100"
    match_flags = 16
    set_flags   = 4
  }
}

The only reason I had set either set_flags or match_flags values was because terraform wanted to update those values every time I ran terraform plan/apply. A better solution for now seems to be to use a lifecycle block to ignore those values altogether.

The other thing terraform wanted to do was remove the as_string configuration from my fortios_router_bgp resource.

      - as_string                          = "<my AS>" -> null
        id                                 = "RouterBgp"
        # (51 unchanged attributes hidden)

        # (20 unchanged blocks hidden)
    }

Although I have an as configured in terraform code and in the Fortigate configuration, I don't have an as_string set in either, nor is that field even configurable. Again, I'll use the lifecycle block for now to ignore this field.

[MaxxLiu22]

Hi all,

Terraform FOS 1.21.1 has been released, addressing several unexpected change issues. The netmask incompatibility for IP arguments should now be resolved. Additionally, the argument prefix may require special handling @zapotah , which I will report to the development team.

Please note that the argument as may need to be adjusted depending on your FOS version—it can be either an integer or a string. Terraform has introduced as and as_string options to accommodate this change. In @ebarrett-Ocient’s case, it appears that as is a string in your FOS version, so you may need to replace as with as_string for compatibility.

Another customer encountered a similar issue, which you can review here. let me know if you still have issues.

Thanks, Maxx

[MaxxLiu22]

Hi @zippanto ,

The argument netmask compatibility issue should be addressed in Terraform FOS 1.22.0. If possible, please feel free to upgrade and see if this resolves the issue.

Thank you for your patience, and don't hesitate to reach out if you have any questions.

Best regards, Maxx

[zapotah]

The new release seems to have solved the prefix cidr endless apply issue. I did not observe any new issues either.