dataplaneapi icon indicating copy to clipboard operation
dataplaneapi copied to clipboard
verified publisher icon haproxytech

Reload behavior of haproxy in AWS EC2 Service Discovery

Open islue opened this issue 2 months ago • 1 comments

HAProxy version 3.2.5-ba0fd6b HAProxy Data Plane API v3.2.3

I'm trying to use haproxy with dataplaneapi to integrate with aws ec2 auto scaling group. I noticed that every time auto scaling group changes, dataplaneapi will apply the changes to backend's servers, write them into haproxy.cfg and reload haproxy.

https://github.com/haproxytech/dataplaneapi/blob/master/discovery/aws_service_discovery_instance.go#L239 -> https://github.com/haproxytech/dataplaneapi/blob/master/discovery/service_discovery_instance.go#L90 -> https://github.com/haproxytech/client-native/blob/master/configuration/service.go#L131 After adding debug loggging to trace code, I think the core of logic to decide whether to reload is the code above.

functions like markRemovedNodes, reorderNodes introduce nodes with modified=true, which would cause reload afterwards. It looks like, scale out won't trigger a reload but scale in will. We heavily use spot instances in auto scaling group and both scale out and in happen very often.

My questions:

  1. Do the behaviors about reload above are by design?
  2. I supposed that reload wouldn't happen when the instance count was less than server_slots_base. Is that right or am I missing anything basically?

cat /etc/dataplaneapi/dataplaneapi.yml

config_version: 2
name: ip-10-1-0-152.ap-northeast-1.compute.internal
dataplaneapi:
  host: 0.0.0.0
  port: 5555
  advertised:
    api_address: ""
    api_port: 0
  scheme:
  - http
  userlist:
    userlist: dataplaneapi
    userlist_file: ""
  transaction:
    transaction_dir: /var/lib/dataplaneapi/transactions
    backups_number: 10
    backups_dir: /var/lib/dataplaneapi/backups
  resources:
    maps_dir: /etc/haproxy/maps
    ssl_certs_dir: /etc/haproxy/ssl
    general_storage_dir: /etc/haproxy/general
    spoe_dir: /etc/haproxy/spoe
haproxy:
  config_file: /etc/haproxy/haproxy.cfg
  haproxy_bin: /usr/sbin/haproxy
  master_runtime: /var/run/haproxy.sock
  master_worker_mode: true
  reload:
    reload_delay: 5
    service_name: haproxy
    reload_strategy: systemd
log_targets:
- log_to: file
  log_file: /var/log/dataplaneapi.log
  log_level: debug
  log_types:
  - access
  - app

cat /etc/haproxy/dataplane/service_discovery/aws.json

{
  "aws_regions": [
    {
      "allowlist": [
        {
          "key": "tag:HAProxy:Service:Name",
          "value": "MyApp"
        },
        {
          "key": "tag:HAProxy:Service:Port",
          "value": "80"
        }
      ],
      "description": "MyApp",
      "enabled": true,
      "id": "8328ca23-c6e4-4e9e-815f-1489baf2baf3",
      "ipv4_address": "private",
      "name": "myapp-service-discovery",
      "region": "ap-northeast-1",
      "retry_timeout": 10,
      "server_slots_base": 500,
      "server_slots_growth_increment": 100,
      "server_slots_growth_type": "linear"
    }
  ]
}

head -n 30 /var/log/dataplaneapi.log

time="2025-10-06T08:36:32Z" level=info msg="HAProxy Data Plane API v3.2.3 daff5f5e"
time="2025-10-06T08:36:32Z" level=info msg="Build from: https://github.com/haproxytech/dataplaneapi"
time="2025-10-06T08:36:32Z" level=info msg="Reload strategy: systemd"
time="2025-10-06T08:36:32Z" level=info msg="Build date: 2025-08-07T14:07:34Z"
time="2025-10-06T08:36:32Z" level=debug msg="discovery job starting" ID=8328ca23-c6e4-4e9e-815f-1489baf2baf3 ServiceDiscovery=AWS
time="2025-10-06T08:36:32Z" level=debug msg="-- command socket updating client_native client"
time="2025-10-06T08:36:33Z" level=info msg="Serving data plane at http://[::]:5555"
time="2025-10-06T08:36:33Z" level=debug msg="-- command socket Server. conf ready..."
time="2025-10-06T08:36:33Z" level=debug msg="-- command socket not set (--debug_socket_path). Not running the command socket server"
time="2025-10-06T08:36:33Z" level=debug msg="Mode: single"
time="2025-10-06T08:36:33Z" level=debug msg="-- command socket handleClientNativeClientUpdate stopped"
time="2025-10-06T08:36:42Z" level=debug msg="discovery job update triggered" ID=8328ca23-c6e4-4e9e-815f-1489baf2baf3 ServiceDiscovery=AWS
time="2025-10-06T08:36:45Z" level=debug msg="discovery job reconciliation completed" ID=8328ca23-c6e4-4e9e-815f-1489baf2baf3 ServiceDiscovery=AWS
time="2025-10-06T08:36:45Z" level=debug msg="Scheduling a new reload..." reload_id=2025-10-06-0
time="2025-10-06T08:36:47Z" level=debug msg="Reload started" reload_id=2025-10-06-0
time="2025-10-06T08:36:48Z" level=debug msg="Reload finished in 253.780028ms" reload_id=2025-10-06-0
time="2025-10-06T08:36:48Z" level=debug msg="Reload successful" reload_id=2025-10-06-0
time="2025-10-06T08:36:48Z" level=debug msg="Handling reload completed, waiting for new requests" reload_id=2025-10-06-0
time="2025-10-06T08:36:55Z" level=debug msg="discovery job update triggered" ID=8328ca23-c6e4-4e9e-815f-1489baf2baf3 ServiceDiscovery=AWS
time="2025-10-06T08:36:57Z" level=debug msg="discovery job reconciliation completed" ID=8328ca23-c6e4-4e9e-815f-1489baf2baf3 ServiceDiscovery=AWS
time="2025-10-06T08:36:57Z" level=debug msg="Scheduling a new reload..." reload_id=2025-10-06-1
time="2025-10-06T08:36:57Z" level=debug msg="Reload started" reload_id=2025-10-06-1
time="2025-10-06T08:36:57Z" level=debug msg="Reload finished in 102.89662ms" reload_id=2025-10-06-1
time="2025-10-06T08:36:57Z" level=debug msg="Reload successful" reload_id=2025-10-06-1
time="2025-10-06T08:36:57Z" level=debug msg="Handling reload completed, waiting for new requests" reload_id=2025-10-06-1
time="2025-10-06T08:37:07Z" level=debug msg="discovery job update triggered" ID=8328ca23-c6e4-4e9e-815f-1489baf2baf3 ServiceDiscovery=AWS
time="2025-10-06T08:37:08Z" level=debug msg="discovery job reconciliation completed" ID=8328ca23-c6e4-4e9e-815f-1489baf2baf3 ServiceDiscovery=AWS
time="2025-10-06T08:37:18Z" level=debug msg="discovery job update triggered" ID=8328ca23-c6e4-4e9e-815f-1489baf2baf3 ServiceDiscovery=AWS
time="2025-10-06T08:37:19Z" level=debug msg="discovery job reconciliation completed" ID=8328ca23-c6e4-4e9e-815f-1489baf2baf3 ServiceDiscovery=AWS
time="2025-10-06T08:37:29Z" level=debug msg="discovery job update triggered" ID=8328ca23-c6e4-4e9e-815f-1489baf2baf3 ServiceDiscovery=AWS

islue avatar Oct 06 '25 12:10 islue

After struggling for a while, I understood that there was no such a runtime api based reload function there. Do you have any develop plan for that, or is it enterprise edition feature?

islue avatar Nov 25 '25 05:11 islue