terraform-provider-fortios
terraform-provider-fortios copied to clipboard
fortinetdev
Error deleting SystemGreTunnel (or VpnIpsecPhase1Interface) resource: Internal Server Error - Internal error when processing the request (500)
I am writing code in Terraform to create a VPN tunnel on a FortiGate (software v7.2.4). Adding always goes successfully however once in a while (once in 7-10 times) I get such errors. When I run this Terraform a second time the removal passes successfully. This is hard to map because it happens completely randomly.
terraform state list output:
fortios_router_multicast.pim-on-gre
fortios_system_gretunnel.gre-tunnel
fortios_vpnipsec_phase1interface.customer_phase1
time_sleep.wait_5_seconds-1
time_sleep.wait_5_seconds-2
time_sleep.wait_5_seconds-3
In debug mode, I found such links to these resources, which were able to throw an error
2024-10-17T12:23:19.034+0200 [WARN] Provider "registry.terraform.io/fortinetdev/fortios" produced an invalid plan for fortios_system_interface.interface-phase1, but we are tolerating it because it is using the legacy plugin SDK. The following problems may be the cause of any confusing errors from downstream operations:
2024-10-17T12:23:19.048+0200 [WARN] Provider "registry.terraform.io/fortinetdev/fortios" produced an invalid plan for fortios_system_interface.interface-tunnel-gre, but we are tolerating it because it is using the legacy plugin SDK. The following problems may be the cause of any confusing errors from downstream operations:
Quick edit: legacy problems were caused by me specifying a /32 mask rather than 255.255.255.255.
Hi @krzysztofmaciejewskiit ,
Thank you so much for bringing up this question. It appears there may be a dependency relationship between your resources. Terraform usually infers dependencies based on the provided configuration. However, if no dependencies are explicitly defined, Terraform will send requests to the FGT in a random and parallel manner. Since the FGT may take varying amounts of time to process each request, this could lead to situations where, for example, an interface is deleted before its related tunnel, resulting in an error.
Would you kindly verify if the dependency relationships are clearly indicated in your configuration? You can define the dependency using the depends_on attribute within your resources or by referencing one resource’s argument in another. Both methods ensure that Terraform processes resources in the correct order, allowing for a smooth creation or deletion workflow where one resource waits for the other to be completed first. let me know if that doesn't solve your question.
resource "fortios_system_interface" "trname" {
name = "vlan_intf"
distance = 5
ip = "54.25.66.3/24"
interface = "port5"
vdom = "root"
type = "vlan"
vlanid = 1550
}
resource "fortios_system_gretunnel" "trname" {
interface = fortios_system_interface.trname.name # using reference to define dependency
ip_version = "4"
local_gw = "3.3.3.3"
local_gw6 = "::"
name = "gretunnel"
remote_gw = "1.1.2.1"
remote_gw6 = "::"
depends_on = [fortios_system_interface.trname] # using depends_on to define dependency
}
destroy fortios_system_interface after completing destroy fortios_system_gretunnel
fortios_system_gretunnel.trname: Destroying... [id=gretunnel]
fortios_system_gretunnel.trname: Destruction complete after 0s
fortios_system_interface.trname: Destroying... [id=vlan_intf]
fortios_system_interface.trname: Destruction complete after 0s
Thanks, Maxx
This is what my code looks like. It controls 'depends_on' so that I have control over what and when gets created and deleted. Two resources ('fortios_system_interface' 'interface-phase1' and 'fortios_system_interface' 'interface-tunnel-gre') have the parameter autogenerated = 'auto', so they don't depend on my own depends_on where I define the delays (creation and destroy), they just depend on their "parent" resources.
#=============== TIMERS ===============#
resource "time_sleep" "wait_5_seconds-1" {
create_duration = "1s"
destroy_duration = "1s"
}
resource "time_sleep" "wait_5_seconds-2" {
depends_on = [time_sleep.wait_5_seconds-1]
create_duration = "1s"
destroy_duration = "1s"
}
resource "time_sleep" "wait_5_seconds-3" {
depends_on = [time_sleep.wait_5_seconds-2]
create_duration = "1s"
destroy_duration = "1s"
}
#======================================#
resource "fortios_vpnipsec_phase1interface" "customer_phase1" {
depends_on = [time_sleep.wait_5_seconds-1]
name = "${var.customer_name}-PHASE1"
interface = var.public_interface_number
local_gw = var.local_public_ip
remote_gw = var.customer_public_ip
psksecret = var.password
ike_version = "2"
keylife = 28800
peertype = "any"
net_device = "disable"
proposal = "aes256-sha256"
dhgrp = "20"
}
resource "fortios_system_interface" "interface-phase1" {
depends_on = [fortios_vpnipsec_phase1interface.customer_phase1]
name = "${var.customer_name}-PHASE1"
interface = var.public_interface_number
ip = "${var.local_public_ip_tunnel} 255.255.255.255"
remote_ip = "${var.customer_public_ip_tunnel} 255.255.255.252"
vdom = var.vdom_name
type = "tunnel"
autogenerated = "auto"
lifecycle { prevent_destroy = true }
}
resource "fortios_vpnipsec_phase2interface" "customer_phase2" {
depends_on = [time_sleep.wait_5_seconds-2]
name = "${var.customer_name}-PHASE2"
phase1name = fortios_vpnipsec_phase1interface.customer_phase1.name
src_subnet = "0.0.0.0 0.0.0.0"
dst_subnet = "0.0.0.0 0.0.0.0"
proposal = "aes256-sha256"
dhgrp = "20"
protocol = 47
keylifeseconds = 3600
}
resource "fortios_system_gretunnel" "gre-tunnel" {
depends_on = [time_sleep.wait_5_seconds-3]
name = "${var.customer_name}-TUN-GRE"
interface = "${var.customer_name}-PHASE1"
local_gw = var.local_public_ip
remote_gw = var.customer_public_ip
}
resource "fortios_system_interface" "interface-tunnel-gre" {
depends_on = [fortios_system_gretunnel.gre-tunnel]
name = "${var.customer_name}-TUN-GRE"
interface = "${var.customer_name}-PHASE1"
ip = "${var.local_public_ip_tunnel} 255.255.255.255"
remote_ip = "${var.customer_public_ip_tunnel} 255.255.255.255"
vdom = var.vdom_name
type = "tunnel"
allowaccess = "ping"
autogenerated = "auto"
lifecycle { prevent_destroy = true }
}
Hi @krzysztofmaciejewskiit ，
Apologies for the delayed response. I reviewed your script, and it looks good to me. I applied and destroyed the resources a few times but didn’t encounter the same issue. However, I noticed a difference in our state files: it seems that your fortios_system_interface resources were not imported into the state file, which means Terraform isn’t managing them. These should be deleted before the GRE tunnel and Phase1 interface. It seems the autogenerated = "auto" function may not be working as expected. Could you please confirm if you're using the latest version of the Terraform FOS provider, 1.21.0? and if your running process is just like what I got here.
root@liang:~/terraform/fgt/github#344# terraform apply -auto-approve
Plan: 8 to add, 0 to change, 0 to destroy.
time_sleep.wait_5_seconds-1: Creating...
time_sleep.wait_5_seconds-1: Creation complete after 1s [id=2024-10-22T22:37:36Z]
time_sleep.wait_5_seconds-2: Creating...
fortios_vpnipsec_phase1interface.customer_phase1: Creating...
fortios_vpnipsec_phase1interface.customer_phase1: Creation complete after 0s [id=cst-PHASE1]
fortios_system_interface.interface-phase1: Creating...
fortios_system_interface.interface-phase1: Creation complete after 0s [id=cst-PHASE1]
time_sleep.wait_5_seconds-2: Creation complete after 1s [id=2024-10-22T22:37:37Z]
time_sleep.wait_5_seconds-3: Creating...
fortios_vpnipsec_phase2interface.customer_phase2: Creating...
fortios_vpnipsec_phase2interface.customer_phase2: Creation complete after 0s [id=cst-PHASE2]
time_sleep.wait_5_seconds-3: Creation complete after 1s [id=2024-10-22T22:37:38Z]
fortios_system_gretunnel.gre-tunnel: Creating...
fortios_system_gretunnel.gre-tunnel: Creation complete after 0s [id=cst-TUN-GRE]
fortios_system_interface.interface-tunnel-gre: Creating...
fortios_system_interface.interface-tunnel-gre: Creation complete after 1s [id=cst-TUN-GRE]
Apply complete! Resources: 8 added, 0 changed, 0 destroyed.
root@liang:~/terraform/fgt/github#344# terraform state list
fortios_system_gretunnel.gre-tunnel
fortios_system_interface.interface-phase1
fortios_system_interface.interface-tunnel-gre
fortios_vpnipsec_phase1interface.customer_phase1
fortios_vpnipsec_phase2interface.customer_phase2
time_sleep.wait_5_seconds-1
time_sleep.wait_5_seconds-2
time_sleep.wait_5_seconds-3
root@liang:~/terraform/fgt/github#344# terraform destroy -auto-approve
Plan: 0 to add, 0 to change, 8 to destroy.
fortios_vpnipsec_phase2interface.customer_phase2: Destroying... [id=cst-PHASE2]
fortios_system_interface.interface-phase1: Destroying... [id=cst-PHASE1]
fortios_system_interface.interface-tunnel-gre: Destroying... [id=cst-TUN-GRE]
fortios_system_interface.interface-phase1: Destruction complete after 0s
fortios_system_interface.interface-tunnel-gre: Destruction complete after 0s
fortios_system_gretunnel.gre-tunnel: Destroying... [id=cst-TUN-GRE]
fortios_vpnipsec_phase2interface.customer_phase2: Destruction complete after 0s
fortios_vpnipsec_phase1interface.customer_phase1: Destroying... [id=cst-PHASE1]
fortios_system_gretunnel.gre-tunnel: Destruction complete after 1s
time_sleep.wait_5_seconds-3: Destroying... [id=2024-10-22T22:37:38Z]
fortios_vpnipsec_phase1interface.customer_phase1: Destruction complete after 1s
time_sleep.wait_5_seconds-3: Destruction complete after 1s
time_sleep.wait_5_seconds-2: Destroying... [id=2024-10-22T22:37:37Z]
time_sleep.wait_5_seconds-2: Destruction complete after 1s
time_sleep.wait_5_seconds-1: Destroying... [id=2024-10-22T22:37:36Z]
time_sleep.wait_5_seconds-1: Destruction complete after 1s
Thanks, Maxx
@MaxxLiu22
Used version of FOS
Initializing the backend...
Initializing provider plugins...
- Reusing previous version of fortinetdev/fortios from the dependency lock file
- Reusing previous version of hashicorp/time from the dependency lock file
- Using previously-installed fortinetdev/fortios v1.21.0
- Using previously-installed hashicorp/time v0.12.1
Terraform has been successfully initialized!
I also updated the Terraform version from 1.8.5 -> 1.9.8.
terraform --version
Terraform v1.9.8
on linux_amd64
+ provider registry.terraform.io/fortinetdev/fortios v1.21.0
+ provider registry.terraform.io/hashicorp/time v0.12.1
I just happened to catch the moment when it ends in failure.
Plan: 0 to add, 0 to change, 23 to destroy.
fortios_vpnipsec_phase2interface.customer_phase2: Destroying... [id=CUSTOMER-PHASE2]
fortios_router_multicast.pim-on-gre-tunnel: Destroying... [id=RouterMulticast]
fortios_router_static.static-route-2: Destroying... [id=3]
fortios_firewall_multicastpolicy.multicast-policy: Destroying... [id=1]
fortios_firewall_policy.unicast-policy-in: Destroying... [id=2]
fortios_router_static.static-route-1: Destroying... [id=2]
fortios_firewall_policy.unicast-policy-out: Destroying... [id=4]
fortios_system_interface.interface-tunnel-gre: Destroying... [id=CUSTOMER-TUN-GRE]
fortios_system_interface.interface-phase1: Destroying... [id=CUSTOMER-PHASE1]
fortios_system_interface.interface-tunnel-gre: Destruction complete after 0s
fortios_system_interface.interface-phase1: Destruction complete after 0s
fortios_system_gretunnel.gre-tunnel: Destroying... [id=CUSTOMER-TUN-GRE]
fortios_vpnipsec_phase2interface.customer_phase2: Destruction complete after 1s
fortios_vpnipsec_phase1interface.customer_phase1: Destroying... [id=CUSTOMER-PHASE1]
fortios_firewall_multicastpolicy.multicast-policy: Destruction complete after 1s
fortios_router_static.static-route-2: Destruction complete after 1s
fortios_router_multicast.pim-on-gre-tunnel: Destruction complete after 1s
fortios_firewall_policy.unicast-policy-out: Destruction complete after 1s
fortios_router_static.static-route-1: Destruction complete after 1s
fortios_router_multicast.pim-on-gre: Destroying... [id=RouterMulticast]
time_sleep.wait_5_seconds-10: Destroying... [id=2024-10-23T05:54:51Z]
fortios_firewall_policy.unicast-policy-in: Destruction complete after 2s
fortios_firewall_address.address: Destroying... [id=CUSTOMER-10.222.1.0/24]
fortios_router_multicast.pim-on-gre: Destruction complete after 1s
fortios_firewall_address.address: Destruction complete after 0s
time_sleep.wait_5_seconds-10: Destruction complete after 1s
time_sleep.wait_5_seconds-9: Destroying... [id=2024-10-23T05:54:50Z]
time_sleep.wait_5_seconds-9: Destruction complete after 1s
time_sleep.wait_5_seconds-8: Destroying... [id=2024-10-23T05:54:49Z]
time_sleep.wait_5_seconds-8: Destruction complete after 1s
time_sleep.wait_5_seconds-7: Destroying... [id=2024-10-23T05:54:48Z]
time_sleep.wait_5_seconds-7: Destruction complete after 1s
time_sleep.wait_5_seconds-6: Destroying... [id=2024-10-23T05:54:47Z]
time_sleep.wait_5_seconds-6: Destruction complete after 1s
time_sleep.wait_5_seconds-5: Destroying... [id=2024-10-23T05:54:46Z]
time_sleep.wait_5_seconds-5: Destruction complete after 1s
time_sleep.wait_5_seconds-4: Destroying... [id=2024-10-23T05:54:45Z]
time_sleep.wait_5_seconds-4: Destruction complete after 1s
╷
│ Error: Error deleting VpnIpsecPhase1Interface resource: Internal Server Error - Internal error when processing the request (500)
│ Cli response:
│ This phase1-interface is currently used
│ command_cli_delete:6722 delete table entry CUSTOMER-PHASE1 unset oper error ret=-23
│ Command fail. Return code -23
│
│
│
╵
╷
│ Error: Error deleting SystemGreTunnel resource: Internal Server Error - Internal error when processing the request (500)
│ Cli response:
│ Can't delete! This interfce is being used.
│ command_cli_delete:6722 delete table entry CUSTOMER-TUN-GRE unset oper error ret=-23
│ Command fail. Return code -23
However, as for your question, I create as you did, that is, with the command “terraform init ; terraform apply -auto-approve”.
Plan: 8 to add, 0 to change, 0 to destroy.
time_sleep.wait_5_seconds-1: Creating...
time_sleep.wait_5_seconds-1: Creation complete after 1s [id=2024-10-23T06:25:44Z]
time_sleep.wait_5_seconds-2: Creating...
fortios_vpnipsec_phase1interface.customer_phase1: Creating...
fortios_vpnipsec_phase1interface.customer_phase1: Creation complete after 0s [id=CUSTOMER-PHASE1]
fortios_system_interface.interface-phase1: Creating...
fortios_system_interface.interface-phase1: Creation complete after 0s [id=CUSTOMER-PHASE1]
time_sleep.wait_5_seconds-2: Creation complete after 1s [id=2024-10-23T06:25:45Z]
time_sleep.wait_5_seconds-3: Creating...
fortios_vpnipsec_phase2interface.customer_phase2: Creating...
fortios_vpnipsec_phase2interface.customer_phase2: Creation complete after 0s [id=CUSTOMER-PHASE2]
time_sleep.wait_5_seconds-3: Creation complete after 1s [id=2024-10-23T06:25:46Z]
fortios_system_gretunnel.gre-tunnel: Creating...
fortios_system_gretunnel.gre-tunnel: Creation complete after 0s [id=CUSTOMER-TUN-GRE]
fortios_system_interface.interface-tunnel-gre: Creating...
fortios_system_interface.interface-tunnel-gre: Creation complete after 1s [id=CUSTOMER-TUN-GRE]
After execution, the list of states looks like this:
terraform state list
fortios_system_gretunnel.gre-tunnel
fortios_system_interface.interface-phase1
fortios_system_interface.interface-tunnel-gre
fortios_vpnipsec_phase1interface.customer_phase1
fortios_vpnipsec_phase2interface.customer_phase2
time_sleep.wait_5_seconds-1
time_sleep.wait_5_seconds-2
time_sleep.wait_5_seconds-3
But with me the deletion looks different (its process), because according to the requirements of the project the deletion must be done by deleting part of the code, rather than executing the command “terraform destroy -auto-approve”. The result (this time without errors):
terraform apply -auto-approve
time_sleep.wait_5_seconds-2: Refreshing state... [id=2024-10-23T06:25:45Z]
time_sleep.wait_5_seconds-3: Refreshing state... [id=2024-10-23T06:25:46Z]
time_sleep.wait_5_seconds-1: Refreshing state... [id=2024-10-23T06:25:44Z]
fortios_system_gretunnel.gre-tunnel: Refreshing state... [id=CUSTOMER-TUN-GRE]
fortios_vpnipsec_phase2interface.customer_phase2: Refreshing state... [id=CUSTOMER-PHASE2]
fortios_vpnipsec_phase1interface.customer_phase1: Refreshing state... [id=CUSTOMER-PHASE1]
fortios_system_interface.interface-phase1: Refreshing state... [id=CUSTOMER-PHASE1]
fortios_system_interface.interface-tunnel-gre: Refreshing state... [id=CUSTOMER-TUN-GRE]
Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
- destroy
...
Plan: 0 to add, 0 to change, 8 to destroy.
fortios_vpnipsec_phase2interface.customer_phase2: Destroying... [id=CUSTOMER-PHASE2]
fortios_system_interface.interface-phase1: Destroying... [id=CUSTOMER-PHASE1]
fortios_system_interface.interface-tunnel-gre: Destroying... [id=CUSTOMER-TUN-GRE]
fortios_system_interface.interface-phase1: Destruction complete after 0s
fortios_system_interface.interface-tunnel-gre: Destruction complete after 0s
fortios_system_gretunnel.gre-tunnel: Destroying... [id=CUSTOMER-TUN-GRE]
fortios_vpnipsec_phase2interface.customer_phase2: Destruction complete after 0s
fortios_vpnipsec_phase1interface.customer_phase1: Destroying... [id=CUSTOMER-PHASE1]
fortios_system_gretunnel.gre-tunnel: Destruction complete after 1s
time_sleep.wait_5_seconds-3: Destroying... [id=2024-10-23T06:25:46Z]
fortios_vpnipsec_phase1interface.customer_phase1: Destruction complete after 1s
time_sleep.wait_5_seconds-3: Destruction complete after 1s
time_sleep.wait_5_seconds-2: Destroying... [id=2024-10-23T06:25:45Z]
time_sleep.wait_5_seconds-2: Destruction complete after 1s
time_sleep.wait_5_seconds-1: Destroying... [id=2024-10-23T06:25:44Z]
time_sleep.wait_5_seconds-1: Destruction complete after 1s
Hi @krzysztofmaciejewskiit,
I hope you’re well! My apologies for the delayed reply, and thank you very much for sharing the details. It looks like Terraform is destroying resources in the expected order, with fortios_system_interface being removed before its dependent resources.
Since you mentioned that the issue appears randomly, it may be influenced by FortiOS performance constraints. Could you kindly try adjusting the parallelism by running:
terraform apply -parallelism=1
terraform destroy -parallelism=1
Terraform default parallelism is 10, this adjustment reduces the number of concurrent tasks, slowing the request rate and potentially allowing FortiOS more time to process each step. If the issue continues, enabling debug mode on FortiOS may provide additional insights for us to review:
diagnose debug cli 8
diagnose debug application httpsd -1
diagnose debug enable
Thank you for your efforts, and please let me know if there’s anything more I can assist with!
Maxx
Hi @MaxxLiu22 I don't see anything interesting in the FortiOS debug, but I fired off the command to debug in Terraform export TF_LOG=DEBUG.
Using the terraform apply command -parallelism=1 -auto-approve also has this problem. I was able to trigger the error again. I attach the errors that occurred.
Please note that I, when deleting a resource, do not execute the terraform destroy command but manually delete a piece of code, save the current code and execute the terraform apply -auto-approve command (Terraform then deletes those resources that were previously created, since they are not in my code.
For your test, I first ran my code using the terraform apply -parallelism=1 -auto-approve command, then removed what I had previously added from the code, saved the file, and then once again used the terraform apply -parallelism=1 -auto-approve command, which detected that what was previously in the code had been removed and Terraform also removed it from its state list.
If you need to know what the structure of my code looks like, I sent it in one of the first messages in this thread.
fortios_vpnipsec_phase1interface.customer_phase1: Destroying... [id=CUSTOMER-PHASE1]
2024-11-06T14:20:37.210+0100 [INFO] Starting apply for fortios_vpnipsec_phase1interface.customer_phase1
2024-11-06T14:20:37.210+0100 [DEBUG] fortios_vpnipsec_phase1interface.customer_phase1: applying the planned Delete change
2024-11-06T14:20:37.653+0100 [INFO] provider.terraform-provider-fortios_v1.21.0: 2024/11/06 14:20:37 FOS-fortios response: {
"http_method":"DELETE",
"revision":"e774dfd61defcddc9e878e90e8af3038",
"revision_changed":false,
"cli_error":"change table entry 'CUSTOMER-PHASE1'\nThis phase1-interface is currently used\ncommand_cli_delete:6722 delete table entry CUSTOMER-PHASE1 unset oper error ret=-23\nCommand fail. Return code -23\ncmd_clean_context 0, abort=0\n",
"error":-23,
"status":"error",
"http_status":500,
"vdom":"root",
"path":"vpn.ipsec",
"name":"phase1-interface",
"mkey":"CUSTOMER-PHASE1",
"serial":"FGVM02TM24013573",
"version":"v7.2.4",
"build":1396
}: timestamp="2024-11-06T14:20:37.653+0100"
2024-11-06T14:20:37.658+0100 [ERROR] provider.terraform-provider-fortios_v1.21.0: Response contains error diagnostic: diagnostic_detail="" tf_proto_version=5.6 tf_resource_type=fortios_vpnipsec_phase1interface tf_rpc=ApplyResourceChange tf_req_id=ad675acd-fdd7-b8ae-492d-d4380b473185 @caller=github.com/hashicorp/[email protected]/tfprotov5/internal/diag/diagnostics.go:58 @module=sdk.proto diagnostic_severity=ERROR
diagnostic_summary=
| Error deleting VpnIpsecPhase1Interface resource: Internal Server Error - Internal error when processing the request (500)
| Cli response:
| change table entry 'CUSTOMER-PHASE1'
| This phase1-interface is currently used
| command_cli_delete:6722 delete table entry CUSTOMER-PHASE1 unset oper error ret=-23
| Command fail. Return code -23
| cmd_clean_context 0, abort=0
tf_provider_addr=provider timestamp="2024-11-06T14:20:37.657+0100"
2024-11-06T14:20:37.674+0100 [DEBUG] State storage *statemgr.Filesystem declined to persist a state snapshot
2024-11-06T14:20:37.674+0100 [ERROR] vertex "fortios_vpnipsec_phase1interface.customer_phase1 (destroy)" error: Error deleting VpnIpsecPhase1Interface resource: Internal Server Error - Internal error when processing the request (500)
Cli response:
change table entry 'CUSTOMER-PHASE1'
This phase1-interface is currently used
command_cli_delete:6722 delete table entry CUSTOMER-PHASE1 unset oper error ret=-23
Command fail. Return code -23
cmd_clean_context 0, abort=0
time_sleep.wait_5_seconds-9: Destroying... [id=2024-11-06T13:20:10Z]
2024-11-06T14:20:37.681+0100 [INFO] Starting apply for time_sleep.wait_5_seconds-9
2024-11-06T14:20:37.682+0100 [DEBUG] time_sleep.wait_5_seconds-9: applying the planned Delete change
time_sleep.wait_5_seconds-9: Destruction complete after 1s
2024-11-06T14:20:38.696+0100 [DEBUG] State storage *statemgr.Filesystem declined to persist a state snapshot
fortios_system_gretunnel.gre-tunnel: Destroying... [id=CUSTOMER-TUN-GRE]
2024-11-06T14:20:38.699+0100 [INFO] Starting apply for fortios_system_gretunnel.gre-tunnel
2024-11-06T14:20:38.699+0100 [DEBUG] fortios_system_gretunnel.gre-tunnel: applying the planned Delete change
2024-11-06T14:20:39.273+0100 [INFO] provider.terraform-provider-fortios_v1.21.0: 2024/11/06 14:20:39 FOS-fortios response: {
"http_method":"DELETE",
"revision":"cd1f63636d04022ec9b7d8e840f583ad",
"revision_changed":true,
"old_revision":"0ecd0e302c3f533d11155d84563196de",
"mkey":"CUSTOMER-TUN-GRE",
"status":"success",
"http_status":200,
"vdom":"root",
"path":"system",
"name":"gre-tunnel",
"serial":"FGVM02TM24013573",
"version":"v7.2.4",
"build":1396
}: timestamp="2024-11-06T14:20:39.273+0100"
fortios_system_gretunnel.gre-tunnel: Destruction complete after 0s
2024-11-06T14:20:39.284+0100 [DEBUG] State storage *statemgr.Filesystem declined to persist a state snapshot
fortios_firewall_address.address: Destroying... [id=CUSTOMER-10.222.1.0/24]
2024-11-06T14:20:39.291+0100 [INFO] Starting apply for fortios_firewall_address.address
2024-11-06T14:20:39.292+0100 [DEBUG] fortios_firewall_address.address: applying the planned Delete change
2024-11-06T14:20:39.797+0100 [INFO] provider.terraform-provider-fortios_v1.21.0: 2024/11/06 14:20:39 FOS-fortios response: {
"http_method":"DELETE",
"revision":"72bc631956cc6057dd37dfc524b20a9e",
"revision_changed":true,
"old_revision":"73a3e6085bb64a2d056d7ea4fdd4922f",
"mkey":"CUSTOMER-10.222.1.0\/24",
"status":"success",
"http_status":200,
"vdom":"root",
"path":"firewall",
"name":"address",
"serial":"FGVM02TM24013573",
"version":"v7.2.4",
"build":1396
}: timestamp="2024-11-06T14:20:39.796+0100"
fortios_firewall_address.address: Destruction complete after 1s
2024-11-06T14:20:39.809+0100 [DEBUG] State storage *statemgr.Filesystem declined to persist a state snapshot
time_sleep.wait_5_seconds-8: Destroying... [id=2024-11-06T13:20:08Z]
2024-11-06T14:20:39.813+0100 [INFO] Starting apply for time_sleep.wait_5_seconds-8
2024-11-06T14:20:39.813+0100 [DEBUG] time_sleep.wait_5_seconds-8: applying the planned Delete change
time_sleep.wait_5_seconds-8: Destruction complete after 1s
2024-11-06T14:20:40.833+0100 [DEBUG] State storage *statemgr.Filesystem declined to persist a state snapshot
time_sleep.wait_5_seconds-7: Destroying... [id=2024-11-06T13:20:06Z]
2024-11-06T14:20:40.836+0100 [INFO] Starting apply for time_sleep.wait_5_seconds-7
2024-11-06T14:20:40.836+0100 [DEBUG] time_sleep.wait_5_seconds-7: applying the planned Delete change
time_sleep.wait_5_seconds-7: Destruction complete after 1s
2024-11-06T14:20:41.846+0100 [DEBUG] State storage *statemgr.Filesystem declined to persist a state snapshot
time_sleep.wait_5_seconds-6: Destroying... [id=2024-11-06T13:20:04Z]
2024-11-06T14:20:41.849+0100 [INFO] Starting apply for time_sleep.wait_5_seconds-6
2024-11-06T14:20:41.849+0100 [DEBUG] time_sleep.wait_5_seconds-6: applying the planned Delete change
time_sleep.wait_5_seconds-6: Destruction complete after 1s
2024-11-06T14:20:42.860+0100 [DEBUG] State storage *statemgr.Filesystem declined to persist a state snapshot
time_sleep.wait_5_seconds-5: Destroying... [id=2024-11-06T13:20:00Z]
2024-11-06T14:20:42.863+0100 [INFO] Starting apply for time_sleep.wait_5_seconds-5
2024-11-06T14:20:42.863+0100 [DEBUG] time_sleep.wait_5_seconds-5: applying the planned Delete change
time_sleep.wait_5_seconds-5: Destruction complete after 1s
2024-11-06T14:20:43.880+0100 [DEBUG] State storage *statemgr.Filesystem declined to persist a state snapshot
time_sleep.wait_5_seconds-4: Destroying... [id=2024-11-06T13:19:58Z]
2024-11-06T14:20:43.883+0100 [INFO] Starting apply for time_sleep.wait_5_seconds-4
2024-11-06T14:20:43.883+0100 [DEBUG] time_sleep.wait_5_seconds-4: applying the planned Delete change
time_sleep.wait_5_seconds-4: Destruction complete after 1s
2024-11-06T14:20:44.902+0100 [DEBUG] State storage *statemgr.Filesystem declined to persist a state snapshot
time_sleep.wait_5_seconds-3: Destroying... [id=2024-11-06T13:19:57Z]
2024-11-06T14:20:44.905+0100 [INFO] Starting apply for time_sleep.wait_5_seconds-3
2024-11-06T14:20:44.905+0100 [DEBUG] time_sleep.wait_5_seconds-3: applying the planned Delete change
time_sleep.wait_5_seconds-3: Destruction complete after 1s
2024-11-06T14:20:45.918+0100 [DEBUG] State storage *statemgr.Filesystem declined to persist a state snapshot
time_sleep.wait_5_seconds-2: Destroying... [id=2024-11-06T13:19:53Z]
2024-11-06T14:20:45.920+0100 [INFO] Starting apply for time_sleep.wait_5_seconds-2
2024-11-06T14:20:45.921+0100 [DEBUG] time_sleep.wait_5_seconds-2: applying the planned Delete change
time_sleep.wait_5_seconds-2: Destruction complete after 1s
2024-11-06T14:20:46.932+0100 [DEBUG] State storage *statemgr.Filesystem declined to persist a state snapshot
╷
│ Error: Error deleting VpnIpsecPhase1Interface resource: Internal Server Error - Internal error when processing the request (500)
│ Cli response:
│ change table entry 'CUSTOMER-PHASE1'
│ This phase1-interface is currently used
│ command_cli_delete:6722 delete table entry CUSTOMER-PHASE1 unset oper error ret=-23
│ Command fail. Return code -23
│ cmd_clean_context 0, abort=0
│
│
│
╵
2024-11-06T14:20:46.943+0100 [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = error reading from server: EOF"
2024-11-06T14:20:46.943+0100 [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = error reading from server: EOF"
2024-11-06T14:20:46.948+0100 [INFO] provider: plugin process exited: plugin=.terraform/providers/registry.terraform.io/hashicorp/time/0.12.1/linux_amd64/terraform-provider-time_v0.12.1_x5 id=2397945
2024-11-06T14:20:46.948+0100 [DEBUG] provider: plugin exited
2024-11-06T14:20:46.959+0100 [INFO] provider: plugin process exited: plugin=.terraform/providers/registry.terraform.io/fortinetdev/fortios/1.21.0/linux_amd64/terraform-provider-fortios_v1.21.0 id=2397954
2024-11-06T14:20:46.959+0100 [DEBUG] provider: plugin exited
Hi @krzysztofmaciejewskiit ,
Sorry for the late reply. I still don't have a clear explanation for this issue. I did comment out the intended deleted resources and used terraform apply instead of terraform destroy, but I still couldn’t reproduce the issue using the provided code. I can trigger the error when I intentionally delete fortios_vpnipsec_phase1interface alone, knowing that fortios_system_interface.interface-phase1 is using it. After applying the entire configuration to the FGT, I checked and found three resources (created by the Terraform files) that refer to VpnIpsecPhase1Interface through the GUI. I wonder if you might have additional resources referring to it without defining a dependency relationship? Also, could you please let me know your CPU count and memory? I suspect this could be another factor contributing to the issue.
resource "fortios_vpnipsec_phase1interface" "customer_phase1" {
depends_on = [time_sleep.wait_5_seconds-1]
name = "${var.customer_name}-PHASE1"
interface = var.public_interface_number
local_gw = var.local_public_ip
remote_gw = var.customer_public_ip
psksecret = var.password
ike_version = "2"
keylife = 28800
peertype = "any"
net_device = "disable"
proposal = "aes256-sha256"
dhgrp = "20"
}
resource "fortios_system_interface" "interface-phase1" {
depends_on = [fortios_vpnipsec_phase1interface.customer_phase1]
name = "${var.customer_name}-PHASE1"
interface = var.public_interface_number
ip = "${var.local_public_ip_tunnel} 255.255.255.255"
remote_ip = "${var.customer_public_ip_tunnel} 255.255.255.252"
vdom = var.vdom_name
type = "tunnel"
autogenerated = "auto"
# lifecycle { prevent_destroy = true }
}
resource "fortios_vpnipsec_phase2interface" "customer_phase2" {
depends_on = [time_sleep.wait_5_seconds-2]
name = "${var.customer_name}-PHASE2"
phase1name = fortios_vpnipsec_phase1interface.customer_phase1.name
src_subnet = "0.0.0.0 0.0.0.0"
dst_subnet = "0.0.0.0 0.0.0.0"
proposal = "aes256-sha256"
dhgrp = "20"
protocol = 47
keylifeseconds = 3600
}
resource "fortios_system_gretunnel" "gre-tunnel" {
depends_on = [time_sleep.wait_5_seconds-3, fortios_system_interface.interface-phase1]
name = "${var.customer_name}-TUN-GRE"
interface = "${var.customer_name}-PHASE1"
local_gw = var.local_public_ip
remote_gw = var.customer_public_ip
}
resource "fortios_system_interface" "interface-tunnel-gre" {
depends_on = [fortios_system_gretunnel.gre-tunnel]
name = "${var.customer_name}-TUN-GRE"
interface = "${var.customer_name}-PHASE1"
ip = "${var.local_public_ip_tunnel2} 255.255.255.255"
remote_ip = "${var.customer_public_ip_tunnel} 255.255.255.255"
vdom = var.vdom_name
type = "tunnel"
allowaccess = "ping"
autogenerated = "auto"
# lifecycle { prevent_destroy = true }
}
FGVMULTM24003288 # get system status
...
License Status: Valid
License Expiration Date: 2025-11-06
VM Resources: 4 CPU, 12008 MB RAM
Log hard disk: Available
...
Thanks, Maxx
@MaxxLiu22
vpn-tunel-1 # get system status
License Status: Valid
License Expiration Date: 2024-12-03
VM Resources: 2 CPU/2 allowed, 16039 MB RAM
Log hard disk: Not available
Hostname: vpn-tunel-1
Current HA mode: standalone
FortiOS x86-64: Yes
We have an additional configuration, but in my opinion it should not directly affect where the error occurs. Are you willing to have a meeting via e.g. Teams? I am open to such a meeting where I will show you exactly the problem. It would be nice if it would work out. You can write on private message to me.
Hi @krzysztofmaciejewskiit ,
I apologize for the inconvenience. Would it be possible for you to download and share your FGT configuration with me? This would help ensure that I can fully restore the device to yours state. At the moment, I think I have all the information, but I am still uncertain about the root cause of the issue. However, based on the CLI error, it seems the interface is being used by another object. If I am still unable to resolve the issue, I will be happy to arrange a remote session with you at your convenience.
Thanks, Maxx
@MaxxLiu22
Awesome. I understand that you want to get the FG configuration together with the configuration created by terraform or the FG configuration alone?
Hi @krzysztofmaciejewskiit , Would it be convenient to obtain both configurations? That would be greatly helpful for me to reproduce the issue.
@MaxxLiu22
When I send it on private message or other way (you can suggest) will it be ok?
@MaxxLiu22 I'll send you an email. In the meantime, another bug was found after FG upgrade from 7.2.6 -> 7.4.5. I created another request https://github.com/fortinetdev/terraform-provider-fortios/issues/351