Load balancer server_ips may not be updated if an instance change modify its private ip

[vincentfretin]

trafficstars

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

Terraform v0.13.3 provider.scaleway v1.16.0

Affected Resource(s)

• scaleway_instance_server private_ip
• scaleway_lb_backend_beta server_ips

Terraform Configuration Files

cloud_init.bash file:

#!/bin/bash
apt update -y -qq
apt install nginx -y                                                             
cat > /var/www/html/index.html <<EOF
<h1>${HOSTNAME}</h1>
EOF

main.tf file:

terraform {
  required_version = ">= 0.13, < 0.14"
  required_providers {
    scaleway = {
      source  = "scaleway/scaleway"
      version = "~> 1.16"
    }
  }
}

resource "scaleway_instance_server" "app" {
  count = 2
  name = "tf-srv-app-${count.index}"
  type  = "DEV1-S"
  image = "ubuntu-focal"

  tags = [ "FocalFossa", "AppInstance" ]
  cloud_init = file("${path.module}/cloud_init.bash")
}

resource "scaleway_lb_ip_beta" "ip" {
}

resource "scaleway_lb_beta" "lb" {
  name = "tf-lb-app"
  ip_id = scaleway_lb_ip_beta.ip.id
  region      = "fr-par"
  type        = "LB-S"
}

resource "scaleway_lb_backend_beta" "backend01" {
  lb_id = scaleway_lb_beta.lb.id
  name = "backend01"
  forward_protocol = "http"
  forward_port = "80"
  server_ips = scaleway_instance_server.app[*].private_ip
}

resource "scaleway_lb_frontend_beta" "frontend01" {
  lb_id = scaleway_lb_beta.lb.id
  backend_id = scaleway_lb_backend_beta.backend01.id
  name = "frontend01"
  inbound_port = "80"
}

outputs.tf file:

output "lb_ip" {
  value = scaleway_lb_beta.lb.ip_address
  description = "Public ip of the load balancer"
}

output "private_ip" {
  value = scaleway_instance_server.app[*].private_ip
  description = "Private ip assigned to the instance"
}

Expected Behavior

scaleway_lb_backend_beta server_ips needs to be updated if instance private ip changed. So terraform plan/apply should always shows the following:

      ~ server_ips               = [
          - "10.70.76.37",
          - "10.70.76.43",
        ] -> (known after apply)

if private ips can potentially be changed.

Actual Behavior

terraform plan/apply says there will be no required change for scaleway_lb_backend_beta server_ips but this is not true.

Steps to Reproduce

First time:

terraform apply

Load balancer is working fine

watch curl --silent http://<app_lb_ip>/

alternate between tf-srv-app-0 and tf-srv-app-1.

Now change the tags of scaleway_instance_server and run terraform apply The 2 instances will be changed (rebooted) and may some times have a different private ip assigned, but the scaleway_lb_backend_beta server_ips will still reference the old private ips. You will see in this case Service Unavailable with the watch curl --silent http://<app_lb_ip>/ command.

To be sure the private ips didn't change, I need to run terraform apply again, if I see no changes, all is good. If there are changes in scaleway_lb_backend_beta server_ips, I need to apply the changes.

References

• similar issue with private ip that changed #529

[vincentfretin]

Another issue, if I change instance_type from DEV1-M to DEV1-S, I got the following plan (which is good):

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  ~ update in-place
+/- create replacement and then destroy

Terraform will perform the following actions:

  # module.app.scaleway_instance_server.app[0] must be replaced
+/- resource "scaleway_instance_server" "app" {
      - additional_volume_ids            = [] -> null
      ~ boot_type                        = "local" -> (known after apply)
        cloud_init                       = <<~EOT
            #!/bin/bash
            apt update -y -qq
            apt install nginx -y
            cat > /var/www/html/index.html <<EOF
            <h1>${HOSTNAME}</h1>
            EOF
        EOT
        disable_dynamic_ip               = false
        disable_public_ip                = false
        enable_dynamic_ip                = false
        enable_ipv6                      = false
      ~ id                               = "fr-par-1/4a840851-7a2d-4db7-9433-1cc5e27c15aa" -> (known after apply)
        image                            = "ubuntu-focal"
      + ipv6_address                     = (known after apply)
      + ipv6_gateway                     = (known after apply)
      ~ ipv6_prefix_length               = 0 -> (known after apply)
        name                             = "tf-sandbox-srv-app-0"
      + organization_id                  = (known after apply)
      ~ private_ip                       = "10.70.76.61" -> (known after apply)
      + public_ip                        = (known after apply)
        security_group_id                = "fr-par-1/7eb2a36b-7d18-498d-a47d-9908d876e3b3"
        state                            = "started"
        tags                             = [
            "FocalFossa",
            "AppInstance",
        ]
      ~ type                             = "DEV1-M" -> "DEV1-S" # forces replacement
      ~ zone                             = "fr-par-1" -> (known after apply)

      ~ root_volume {
            delete_on_termination = true
          ~ size_in_gb            = 40 -> (known after apply)
          ~ volume_id             = "fr-par-1/3520162f-dd88-4029-8cd3-4edf9fc20111" -> (known after apply)
        }
    }

  # module.app.scaleway_instance_server.app[1] must be replaced
+/- resource "scaleway_instance_server" "app" {
      - additional_volume_ids            = [] -> null
      ~ boot_type                        = "local" -> (known after apply)
        cloud_init                       = <<~EOT
            #!/bin/bash
            apt update -y -qq
            apt install nginx -y
            #apt upgrade -y -qq
            cat > /var/www/html/index.html <<EOF
            <h1>${HOSTNAME}</h1>
            EOF
        EOT
        disable_dynamic_ip               = false
        disable_public_ip                = false
        enable_dynamic_ip                = false
        enable_ipv6                      = false
      ~ id                               = "fr-par-1/a1d5e9c6-6c03-43c8-a14e-563aec25fa27" -> (known after apply)
        image                            = "ubuntu-focal"
      + ipv6_address                     = (known after apply)
      + ipv6_gateway                     = (known after apply)
      ~ ipv6_prefix_length               = 0 -> (known after apply)
        name                             = "tf-sandbox-srv-app-1"
      + organization_id                  = (known after apply)
      ~ private_ip                       = "10.70.76.55" -> (known after apply)
      + public_ip                        = (known after apply)
        security_group_id                = "fr-par-1/7eb2a36b-7d18-498d-a47d-9908d876e3b3"
        state                            = "started"
        tags                             = [
            "FocalFossa",
            "AppInstance",
        ]
      ~ type                             = "DEV1-M" -> "DEV1-S" # forces replacement
      ~ zone                             = "fr-par-1" -> (known after apply)

      ~ root_volume {
            delete_on_termination = true
          ~ size_in_gb            = 40 -> (known after apply)
          ~ volume_id             = "fr-par-1/120e29b6-4902-45ae-9a28-0b3eb805012b" -> (known after apply)
        }
    }

  # module.app.scaleway_lb_backend_beta.backend01 will be updated in-place
  ~ resource "scaleway_lb_backend_beta" "backend01" {
        forward_port             = 80
        forward_port_algorithm   = "roundrobin"
        forward_protocol         = "http"
        health_check_delay       = "1m0s"
        health_check_max_retries = 2
        health_check_port        = 80
        health_check_timeout     = "30s"
        id                       = "fr-par/2739ce27-006b-4284-b5f9-9643024d2680"
        lb_id                    = "fr-par/0188bc7c-9d8c-4923-802f-ac7b6b61a6f5"
        name                     = "backend01"
        on_marked_down_action    = "none"
        send_proxy_v2            = false
      ~ server_ips               = [
          - "10.70.76.61",
          - "10.70.76.55",
        ] -> (known after apply)
        sticky_sessions          = "none"

        health_check_tcp {}
    }

Plan: 2 to add, 1 to change, 2 to destroy.

Changes to Outputs:
  ~ app_private_ip                       = [
      - "10.70.76.61",
      - "10.70.76.55",
      + (known after apply),
      + (known after apply),
    ]

Apply complete! Resources: 2 added, 1 changed, 2 destroyed.

Outputs:

app_lb_ip = [
  "x.x.x.x",
]

app_private_ip = [
  "10.70.76.37",
  "10.70.76.43",
]

The server_ips did change in the terraform.tfstate but it didn't change on the load balancer when I look into the web console. It still references the old private ips.

[vincentfretin]

Ah the server ips really changed in the load balancer, there seems to be a cache issue in the Overview panel in the console. In the backends tab, the server ips are correct. Backends are not responding though because nginx is not installed, some issue with the cloud-init script that wasn't executed apparently, it may be my fault for this one, anyway destroy and apply fixed the issue.

[vincentfretin]

If creating the load balancer and one instance used in the server_ips is marked as changed, the following can happen:

Error: Provider produced inconsistent final plan

When expanding the plan for
scaleway_lb_backend_beta.backend01 to include new values
learned so far during apply, provider
"registry.terraform.io/scaleway/scaleway" produced an invalid new value for
.server_ips[0]: was cty.StringVal("10.69.130.95"), but now
cty.StringVal("10.69.70.1").

This is a bug in the provider, which should be reported in the provider's own
issue tracker.

So for now, I will apply changes to the instances first, then add the load balancer. :)

[vincentfretin]

@remyleone @Sh4d1 About the issue of the "Load balancers" tab in the web console showing old server ips for a load balancer I described above with my screenshots, do you have another tracker for web console issues to track it?

[remyleone]

For the bug in the console, I would suggest opening a ticket on your scaleway account with the given screenshot and a description of the problem.

[vincentfretin]

Ok, I just created a support ticket. I tried to reproduce the issue creating an instance and load balancer via the web console. I see two ips for Server IPs on the load balancer from the Load balancers tab, that is different for the single server ip I added when I created the load balancer. Maybe it's not a bug? Is Server IPs showed in the load balancer here something different that a backend server ips? Anyway it's confusing. I put that in the support ticket. :)

[vincentfretin]

I got a discussion on the support ticket. So this was a misunderstanding of my part, the Server IPs showed in the "Load balancers" tab is the two internal ips of the load balancer, something completely different from the backend server ips. I proposed to rename this label to something like "Load balancer internal ips" to the Front-End team.

[Monitob]

I'm unavailable to reproduce the issue described here. It looks fixed to me for other tickets on the resource instances. Please in order to fix an IP or use the dynamic IP reservations please check this example