haproxy
haproxy copied to clipboard
haproxy
Documentation: Clarify the meaning of 'hold' in the 'resolvers' section
Detailed Description of the Problem
The documentation describes the hold directive in a resolvers sections as:
hold <status> <period>
Defines <period> during which the last name resolution should be kept based
on last resolution <status>
<status> : last name resolution status. Acceptable values are "nx",
"other", "refused", "[timeout](https://cbonte.github.io/haproxy-dconv/2.5/configuration.html#)", "valid", "obsolete".
<period> : interval between two successive name resolution when the last
answer was in <status>. It follows the HAProxy time format.
<period> is in milliseconds by default.
This description has led some users to believe that HAProxy caches DNS responses and that hold sets a TTL for expiring that cache entry in HAProxy. For example, one might think that hold valid 30s will cache the DNS response for 30 seconds, causing HAProxy to only query the DNS server again after that time.
However, the meaning, as I understand it, is actually that the up/down status of the server will be held for 30 seconds.
In my own notes, I have described it this way:
How long to "hold" a backend server's up/down status depending on the name resolution status.
For example, if an NXDOMAIN response is returned, keep the backend server in its current state (up) for
at least another 30 seconds (`hold nx 30s`) before marking it as down due to DNS not having a record for it.
If I have accurately described it, then I think the official documentation should be clarified so that users don't believe hold means a DNS cache TTL.
I am still unclear on the use case of hold valid 10s...is its purpose for how long to maintain a server's down status before bringing it back up after receiving a valid DNS response?
Expected Behavior
I was confused myself about this directive, but I have also seen other users express the same confusion. I discovered it because HAProxy was sending DNS queries more often than I expected, since I had set a long hold period.
Steps to Reproduce the Behavior
Not applicable
Do you have any idea what may have caused this?
No response
Do you have an idea how to solve the issue?
I recommend changing the description for hold to:
hold <status> <period>
Defines a <period> for which the backend server that relies on the DNS query for address resolution will keep its current up/down state when the DNS response is <status>. For example, if "hold nx 30s" and the DNS response is NXDOMAIN, maintain the server's up state for another 30 seconds before marking it as down due to the DNS being unable to resolve the domain.
<status> : last name resolution status. Acceptable values are "nx",
"other", "refused", "timeout", "valid", "obsolete".
<period> : interval between two successive name resolution when the last
answer was in <status>. It follows the HAProxy time format.
<period> is in milliseconds by default.
What is your configuration?
Here is a `resolvers` section:
resolvers mynameservers
nameserver ns1 192.168.2.10:53
nameserver ns2 192.168.3.10:53
accepted_payload_size 512
parse-resolv-conf
hold valid 10s
hold other 30s
hold refused 30s
hold nx 30s
hold timeout 30s
hold obsolete 30s
resolve_retries 3
timeout retry 1s
timeout resolve 1s
### Output of `haproxy -vv`
```plain
$ haproxy -vv
HAProxy version 2.6-dev9-314e6e-13 2022/05/11 - https://haproxy.org/
Status: development branch - not safe for use in production.
Known bugs: https://github.com/haproxy/haproxy/issues?q=is:issue+is:open
Running on: Linux 4.15.0-176-generic #185-Ubuntu SMP Tue Mar 29 17:40:04 UTC 2022 x86_64
Build options :
TARGET = linux-glibc
CPU = generic
CC = cc
CFLAGS = -O2 -g -Wall -Wextra -Wundef -Wdeclaration-after-statement -Wfatal-errors -Wtype-limits -Wshift-negative-value -Wshift-overflow=2 -Wduplicated-cond -Wnull-dereference -fwrapv -Wno-address-of-packed-member -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-clobbered -Wno-missing-field-initializers -Wno-cast-function-type -Wno-string-plus-int -Wno-atomic-alignment
OPTIONS = USE_PCRE=1 USE_OPENSSL=1 USE_LUA=1 USE_ZLIB=1 USE_SYSTEMD=1 USE_PROMEX=1
DEBUG = -DDEBUG_STRICT -DDEBUG_MEMORY_POOLS
Feature list : +EPOLL -KQUEUE +NETFILTER +PCRE -PCRE_JIT -PCRE2 -PCRE2_JIT +POLL +THREAD +BACKTRACE -STATIC_PCRE -STATIC_PCRE2 +TPROXY +LINUX_TPROXY +LINUX_SPLICE +LIBCRYPT +CRYPT_H -ENGINE +GETADDRINFO +OPENSSL +LUA +ACCEPT4 -CLOSEFROM +ZLIB -SLZ +CPU_AFFINITY +TFO +NS +DL +RT -DEVICEATLAS -51DEGREES -WURFL +SYSTEMD -OBSOLETE_LINKER +PRCTL -PROCCTL +THREAD_DUMP -EVPORTS -OT -QUIC +PROMEX -MEMORY_PROFILING
Default settings :
bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
Built with multi-threading support (MAX_THREADS=64, default=2).
Built with OpenSSL version : OpenSSL 1.1.1 11 Sep 2018
Running on OpenSSL version : OpenSSL 1.1.1 11 Sep 2018
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
Built with Lua version : Lua 5.3.3
Built with the Prometheus exporter as a service
Built with network namespace support.
Built with zlib version : 1.2.11
Running on zlib version : 1.2.11
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Support for malloc_trim() is enabled.
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with PCRE version : 8.39 2016-06-14
Running on PCRE version : 8.39 2016-06-14
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Encrypted password support via crypt(3): yes
Built with gcc compiler version 7.5.0
Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.
Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
h2 : mode=HTTP side=FE|BE mux=H2 flags=HTX|HOL_RISK|NO_UPG
fcgi : mode=HTTP side=BE mux=FCGI flags=HTX|HOL_RISK|NO_UPG
<default> : mode=HTTP side=FE|BE mux=H1 flags=HTX
h1 : mode=HTTP side=FE|BE mux=H1 flags=HTX|NO_UPG
<default> : mode=TCP side=FE|BE mux=PASS flags=
none : mode=TCP side=FE|BE mux=PASS flags=NO_UPG
Available services : prometheus-exporter
Available filters :
[CACHE] cache
[COMP] compression
[FCGI] fcgi-app
[SPOE] spoe
[TRACE] trace
### Last Outputs and Backtraces
_No response_
### Additional Information
_No response_
Hi there, I think @NickMRamirez is correct and his description describes better the behavior of the resolver.
Just wanted to confirm I just spent about 15-30 minutes trying to confirm what was happening, so I fully support this update to the docs. Thanks @NickMRamirez. :)
To further clarify, could you answer the detail of what happens when a server is down due to a failure? For example, given:
hold timeout 60s
hold valid 10s
If the server is down after 61 seconds of timeout status and then receives a valid response, does it immediately flip to UP or does it wait for 10s?
@NickMRamirez, your description is only true for UP->DOWN updates. But DOWN->UP updates are immediately performed when a valid response is received. And your right for all hold status except for hold.obsolete and hold.valid. hold.valid is different than others because it only says how long a valid response can be used by do-resolve action without triggering a new resolution. However it does not affect dynamic resolutions for servers. hold.obsolete is only used to remove obsolete items. It is only checked when a response containing at least an answer item is received. So it is not evaluated on NX responses or on timeout. If an item is considered as obsolete, it is removed. And, at this stage, the server state is only changed for SRV records. When a A/AAAA item is considered as obsolete, the server state is not changed. Then the result of the valid response is applied. It explains why it takes hold.obsolete seconds to change a server IP and why, when a SRV record is updated, there is a UP -> DOWN -> UP. Sorry it is probably a bit foggy but it is really a mess to describe :)
For others hold status, your description should probably be updated too. Because, based on your example, it suggests (at least it is my understanding :) that when a NX response is received, HAProxy will wait 30sec before setting the server to DOWN. In fact, the server is set to DOWN when the NX response is received if no valid response was received on the last 30sec. The action is not delayed, it is performed when the NX response is received.
@lseelenbinder, with your configuration, a server will be set to DOWN if a response timeout occurs when no valid response was received on the last 60sec. If a valid response is received, it is immediately set to UP. hold.valid is not used for server resolution.
@capflam For that last part, I think we are saying something similar. You said:
In fact, the server is set to DOWN when the NX response is received if no valid response was received on the last 30sec.
UPDATE on my comment: Checking my understanding -- In that case, hold will depend wholly on how often name resolutions occur, as set by timeout resolve. If hold is looking backwards in time, then it must receive at least two NX responses within 30 seconds, for example.
I suppose this is a true statement:
<period> defines the subsequent span of time during which a NOERROR
response must be received, or else <status> will cause the server to be
taken out of the load balancing rotation.
The idea being that it doesn't matter if you look back 30 seconds or forward 30 seconds. It only matters that you have a span of 30 seconds with no good (NOERROR) responses within it. Whether the point of reference is at the beginning of that span or the end of it, shouldn't matter?
Well, it matters because only the last error code is processed, when it is received, not after the <period> delay. Previous error codes are not saved. For instance, with these timeouts:
hold nx 30s
hold timeout 300s
If you receive a first NXDOMAIN 1 second after the last valid resolution, the server status is not updated and this is expected. 30 seconds after the last valid resolution, if you receive a NXDOMAIN, the server will be set to DOWN. But if, instead of a NXDOMAIN, the resolution timed out, nothing happens because you should wait 300s to switch a server to DOWN on a timeout error. The previous NXDOMAIN error is lost. When an error is received, there is no timer. So I guess the doc should make it clear. The error is evaluated when it is received, comparing the current date against the date of the last valid resolution. If the delta exceeds <period>, the server is set to DOWN. Otherwise, the error is ignored.
Reading your proposal, I understand that an error is processed after the '<period> delay if no valid response is received in the mean time and independently on other errors. It is not pretty clear, at least for me, that the same error must be received again after a <period> delay (or more) to be really processed.
Adding @bedis to the discussion to know what to put in the doc regarding this. In any case it's clear that this needs to be improved but we must be sure to write something realistic.
I have revised my proposed definition to take into account the points raised by @capflam. How does this look?
hold <status> <period>
Upon receiving the DNS response <status>, determines whether a server's state
should change from UP to DOWN. To make that determination, it checks whether
any valid status has been received during the past <period> in order to counteract
the just received invalid status.
<status> : last name resolution status.
nx After receiving an NXDOMAIN status, check for any valid
status during the concluding period.
refused After receiving a REFUSED status, check for any valid
status during the concluding period.
timeout After the "timeout retry" has struck, check for any
valid status during the concluding period.
other After receiving any other invalid status, check for any
valid status during the concluding period.
valid Applies only to "http-request do-resolve" and
"tcp-request content do-resolve" actions. It defines the
period for which the server will maintain a valid response
before triggering another resolution. It does not affect
dynamic resolution of servers.
obsolete Defines how long to wait before removing obsolete DNS
records after an updated answer record is received. It
applies to SRV records.
<period> : Amount of time into the past during which a valid response must have
been received. It follows the HAProxy time format and is in milliseconds
by default.
For a server that relies on dynamic DNS resolution to determine its IP address,
receiving an invalid DNS response, such as NXDOMAIN, will lead to changing the
server's state from UP to DOWN. The hold directives define how far into the past
to look for a valid response. If a valid response has been received within <period>,
the just received invalid status will be ignored.
Unless a valid response has been receiving during the concluding period, the server
will be marked as DOWN. For example, if "hold nx 30s" is set and the last received DNS
response was NXDOMAIN, the server will be marked DOWN unless a valid response has been
received during the last 30 seconds.
A server in the DOWN state will be marked UP immediately upon receiving a valid status
from the DNS server.
A separate behavior exists for "hold valid" and "hold obsolete".
It seems good to me. Thanks @NickMRamirez, I will update the documentation.
Now pushed. The patch will be backported with other fixes as usual. Many thanks @NickMRamirez !