NetworkConfiguration
dhcpcd program crash in case flooded with ICMPv6 RAs (Router Advertisements)
Hi,
in our robustness testing, our system got flooded with ICMPv6 RAs (Router Advertisements). Problem happens on dhcpcd version 10.2.2
See from syslog:
Jun 25 09:07:33 air6419_mongoose dhcpcd[8455]: port0: Router Advertisement from fe80::551 Jun 25 09:07:33 air6419_mongoose dhcpcd[8455]: port0: adding address 2001:0:c:4631:3b9c:11ff:a1b0:c45d/64 Jun 25 09:07:33 air6419_mongoose dhcpcd[8455]: port0: adding route to 2001:0:c:4631::/64 Jun 25 09:07:33 air6419_mongoose dhcpcd[8455]: port0: Router Advertisement from fe80::553 Jun 25 09:07:33 air6419_mongoose dhcpcd[8455]: port0: adding address 2001:0:c:4632:3b9c:11ff:a1b0:c45d/64 Jun 25 09:07:33 air6419_mongoose dhcpcd[8455]: port0: adding route to 2001:0:c:4632::/64 Jun 25 09:07:33 air6419_mongoose dhcpcd[8455]: port0: Router Advertisement from fe80::555 Jun 25 09:07:33 air6419_mongoose dhcpcd[8455]: port0: adding address 2001:0:c:4633:3b9c:11ff:a1b0:c45d/64 Jun 25 09:07:33 air6419_mongoose dhcpcd[8455]: port0: adding route to 2001:0:c:4633::/64 Jun 25 09:07:33 air6419_mongoose dhcpcd[8455]: dhcpcd_handlelink: unexpected event 0x0101 Jun 25 09:07:33 air6419_mongoose dhcpcd[8455]: route socket overflowed (rcvbuflen 106496) - learning interface state Jun 25 09:07:33 air6419_mongoose dhcpcd[8455]: drained 215 messages
Each RA triggers dhcpcd to reconfigure routes and addresse, and we get in this program crash due to memory route socket overflowed. See backtrace bellow:
Core was generated by `dhcpcd: [privileged proxy] port0 [ip4] [ip6] '.
Program terminated with signal SIGABRT, Aborted.
#0 __pthread_kill_implementation (threadid=
[**ALERT: The abort() might not be exactly invoked from the following function line. If the trail function contains multiple abort() calls, then you should cross check by other means to get correct abort() call location. This is due to the optimized compilation which hides the debug info for multiple abort() calls in a given function. Refer TR HU16995 for more information]
#4 0x0000ffff8382d4b0 in __assert_fail_base (fmt=0xffff83946370 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=assertion@entry=0xaaaae068c368 "(bufp - 1) == '\0'", file=file@entry=0xaaaae068c340 "script.c", line=line@entry=198, function=function@entry=0xaaaae0690188 <PRETTY_FUNCTION.4> "script_buftoenv") at assert.c:96
#5 0x0000ffff8382d530 in __assert_fail (assertion=assertion@entry=0xaaaae068c368 "(bufp - 1) == '\0'", file=file@entry=0xaaaae068c340 "script.c", line=line@entry=198, function=function@entry=0xaaaae0690188 <PRETTY_FUNCTION.4> "script_buftoenv") at assert.c:105
#6 0x0000aaaae0660d90 in script_buftoenv (ctx=ctx@entry=0xffffd436aa38, buf=buf@entry=0xffffd435a898 "PATH=/usr/bin:/usr/sbin:/bin:/sbin", len=len@entry=65696) at script.c:198
#7 0x0000aaaae0682384 in ps_root_run_script (len=65696, data=0xffffd435a898, ctx=0xffffd436aa38) at privsep-root.c:291
#8 ps_root_recvmsgcb (arg=0xffffd436aa38, psm=0xffffd435a858, msg=0xffffd435a810) at privsep-root.c:556
#9 0x0000aaaae0680f4c in ps_recvpsmsg (ctx=0xffffd436aa38, fd=
(gdb) bt full
#0 __pthread_kill_implementation (threadid=
[**ALERT: The abort() might not be exactly invoked from the following function line. If the trail function contains multiple abort() calls, then you should cross check by other means to get correct abort() call location. This is due to the optimized compilation which hides the debug info for multiple abort() calls in a given function. Refer TR HU16995 for more information]
#4 0x0000ffff8382d4b0 in __assert_fail_base (fmt=0xffff83946370 "%s%s%s:%u: %s%sAssertion %s' failed.\n%n", assertion=assertion@entry=0xaaaae068c368 "*(bufp - 1) == '\\0'", file=file@entry=0xaaaae068c340 "script.c", line=line@entry=198, function=function@entry=0xaaaae0690188 <__PRETTY_FUNCTION__.4> "script_buftoenv") at assert.c:96 str = 0xaaab1fc5cce0 "\\\374\261\252\n" total = 4096 #5 0x0000ffff8382d530 in __assert_fail (assertion=assertion@entry=0xaaaae068c368 "*(bufp - 1) == '\\0'", file=file@entry=0xaaaae068c340 "script.c", line=line@entry=198, function=function@entry=0xaaaae0690188 <__PRETTY_FUNCTION__.4> "script_buftoenv") at assert.c:105 No locals. #6 0x0000aaaae0660d90 in script_buftoenv (ctx=ctx@entry=0xffffd436aa38, buf=buf@entry=0xffffd435a898 "PATH=/usr/bin:/usr/sbin:/bin:/sbin", len=len@entry=65696) at script.c:198 env = <optimized out> envp = <optimized out> bufp = <optimized out> endp = 0xffffd436a938 "" nenv = <optimized out> __PRETTY_FUNCTION__ = "script_buftoenv" #7 0x0000aaaae0682384 in ps_root_run_script (len=65696, data=0xffffd435a898, ctx=0xffffd436aa38) at privsep-root.c:291 envbuf = 0xffffd435a898 "PATH=/usr/bin:/usr/sbin:/bin:/sbin" argv = {0xaaaae068aa38 "/usr/libexec/dhcpcd-run-hooks", 0x0} pid = <optimized out> status = 533228656 envbuf = <optimized out> argv = {<optimized out>, <optimized out>} pid = <optimized out> --Type <RET> for more, q to quit, c to continue without paging-- status = <optimized out> __func__ = "ps_root_run_script" #8 ps_root_recvmsgcb (arg=0xffffd436aa38, psm=0xffffd435a858, msg=0xffffd435a810) at privsep-root.c:556 ctx = 0xffffd436aa38 cmd = <optimized out> psp = <optimized out> iov = <optimized out> data = 0xffffd435a898 rdata = 0x0 len = 65696 rlen = 0 buf = <error reading variable buf (value requires 65696 bytes, which is more than max-value-size)> mtime = 187651949358192 err = <optimized out> free_rdata = false __func__ = "ps_root_recvmsgcb" __PRETTY_FUNCTION__ = "ps_root_recvmsgcb" #9 0x0000aaaae0680f4c in ps_recvpsmsg (ctx=0xffffd436aa38, fd=<optimized out>, events=<optimized out>, callback=callback@entry=0xaaaae0681d60 <ps_root_recvmsgcb>, cbctx=0xffffd436aa38) at privsep.c:1181 psm = <error reading variable psm (value of type ps_msg' requires 65760 bytes, which is more than max-value-size)>
len = 65760
dlen = 65696
iov = {{iov_base = 0xffffd435a898, iov_len = 65696}}
msg = {msg_name = 0x0, msg_namelen = 0, msg_iov = 0xffffd435a848, msg_iovlen = 1, msg_control = 0x0, msg_controllen = 0, msg_flags = 0}
stop = false
func = "ps_recvpsmsg"
#10 0x0000aaaae0681b28 in ps_root_recvmsg (arg=
(gdb) thread apply all bt
Thread 1 (LWP 8455):
#0 __pthread_kill_implementation (threadid=
[**ALERT: The abort() might not be exactly invoked from the following function line. If the trail function contains multiple abort() calls, then you should cross check by other means to get correct abort() call location. This is due to the optimized compilation which hides the debug info for multiple abort() calls in a given function. Refer TR HU16995 for more information]
#4 0x0000ffff8382d4b0 in __assert_fail_base (fmt=0xffff83946370 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=assertion@entry=0xaaaae068c368 "(bufp - 1) == '\0'", file=file@entry=0xaaaae068c340 "script.c", line=line@entry=198, function=function@entry=0xaaaae0690188 <PRETTY_FUNCTION.4> "script_buftoenv") at assert.c:96
#5 0x0000ffff8382d530 in __assert_fail (assertion=assertion@entry=0xaaaae068c368 "(bufp - 1) == '\0'", file=file@entry=0xaaaae068c340 "script.c", line=line@entry=198, function=function@entry=0xaaaae0690188 <PRETTY_FUNCTION.4> "script_buftoenv") at assert.c:105
#6 0x0000aaaae0660d90 in script_buftoenv (ctx=ctx@entry=0xffffd436aa38, buf=buf@entry=0xffffd435a898 "PATH=/usr/bin:/usr/sbin:/bin:/sbin", len=len@entry=65696) at script.c:198
#7 0x0000aaaae0682384 in ps_root_run_script (len=65696, data=0xffffd435a898, ctx=0xffffd436aa38) at privsep-root.c:291
#8 ps_root_recvmsgcb (arg=0xffffd436aa38, psm=0xffffd435a858, msg=0xffffd435a810) at privsep-root.c:556
--Type <RET> for more, q to quit, c to continue without paging--
#9 0x0000aaaae0680f4c in ps_recvpsmsg (ctx=0xffffd436aa38, fd=
Can you please check how we could avoid this crash under ICMPv6 RA fuzzing.
Interesting.
At first glance it seems that we are sending a very big environment to the script to run and there is a NULL termination issue with that. Can you please try compiling out PRIVSEP support and test? I want to know if PRIVSEP is the cause of this crash or not.
Hi, I compiled without PRIVSEP now waiting on test. In meantime I was checking backtrace more. As from backtrace we have:
#7 ps_root_run_script (len=65696, data=...) at privsep-root.c:291 #8 ps_root_recvmsgcb(...) at privsep-root.c:556
And in privsep-root.c:
ps_root_recvmsgcb(...) { switch (cmd) { case PS_SCRIPT: return ps_root_run_script(len, data, ctx);
And we have ps_root_run_script() invoked, it seems cmd was PS_SCRIPT. Right?
The flow of processing seems to be: script_runreason() -> ps_root_script() -> ps_sendcmd(... PS_SCRIPT ...) -> crash happens on the receive msg handling in script_buftoenv().
I suspect we are setting data and len used in script_runreason() inside make_env() at line 553:
ctx->script_buf = buf;
ctx->script_buflen = buf_len;
But below this line we have script_buftoenv() call:
if (script_buftoenv(ctx, ctx->script_buf, (size_t)buf_pos) == NULL)
goto eexit;
if we had wild buffer and not correctly null-terminated it would assert here when called from make_env(), right?
I was thinking to add a secure check:
if (cmd == PS_SCRIPT) { const char *bytes = (const char *)data; if (len == 0 || bytes[len - 1] != '\0') { logerr("%s: PS_SCRIPT data not null-terminated", func); errno = EINVAL; return -1; } }
in ps_sendcmd() like below:
ssize_t ps_sendcmd(struct dhcpcd_ctx *ctx, int fd, uint16_t cmd, unsigned long flags, const void *data, size_t len) { struct ps_msghdr psm = { .ps_cmd = cmd, .ps_flags = flags, }; struct iovec iov[] = { { .iov_base = UNCONST(data), .iov_len = len } }; struct msghdr msg = { .msg_iov = iov, .msg_iovlen = 1, };
if (cmd == PS_SCRIPT) {
const char *bytes = (const char *)data;
if (len == 0 || bytes[len - 1] != '\0') {
logerr("%s: PS_SCRIPT data not null-terminated", __func__);
errno = EINVAL;
return -1;
}
}
return ps_sendpsmmsg(ctx, fd, &psm, &msg);
}
This would prevent crash. What is your thinking?
I suspect we are setting data and len used in script_runreason() inside make_env() at line 553:
So your platform lacks support for open_memstream(3)? That might not be well tested I guess.
Hi, still waiting on test results without PRIVSEP. In the dhcpcd 10.2.2 folder once I downloaded it from github in /dhcpcd-10.2.2/configure We can find: if [ -z "$OPEN_MEMSTREAM" ]; then printf "Testing for open_memstream ... " cat <<EOF >_open_memstream.c #include <stdio.h> int main(void) { return open_memstream(NULL, NULL) != NULL ? 0 : 1; }
and in build log we have: temp/log.do_configure.3166715 49:Testing for open_memstream ... yes
So, we are having OPEN_MEMSTREAM enabled for our plaform.
With further code walkthrough I would propose to do following in the make_env(). Here is my git patch:
From 153f57a091df23100a01efca1b4e69921c141e07 Mon Sep 17 00:00:00 2001 From: "Sime Zupanovic (EXT)" [email protected] Date: Tue, 15 Jul 2025 22:17:10 +0200 Subject: [PATCH] script: ensure script_buf is null-terminated
src/script.c | 12 ++++++++++++ 1 file changed, 12 insertions(+)
diff --git a/src/script.c b/src/script.c index abf62f6..eaea43a 100755 --- a/src/script.c +++ b/src/script.c @@ -544,6 +544,18 @@ dumplease: goto eexit; }
/* Ensure null-termination */
if (fputc('\0', fp) == EOF) {
logerr("%s: failed to write null terminator", __func__);
goto eexit;
}
/* Safety check: ensure the buffer is null-terminated */
if (ctx->script_buf == NULL || ctx->script_buf[buf_pos - 1] != '\0') {
logerrx("%s: script_buf not null-terminated!", __func__);
goto eexit;
}
-- 2.36.0
so in this step we know script_buf is fully written. fflush(fp) has already flushed the buffer. My idea is to add here writing of '\0', to guarantee null-termination of the buffer. What do you think?
regards, Sime
Hi,
so, what I suspect. As we have a flood of ICMPv6 RA packets, this may happen:
make_env() is called repeatedly and quickly. The internal FILE* fp (ctx->script_fp) is reused. If rewind(fp) is used, but ctx->script_buf is reused and fputc('\0') was missing, the new buffer may be truncated, incomplete, or not null-terminated.
Then when we call: script_buftoenv(ctx, ctx->script_buf, buflen); we can have assert failing if buf[buflen - 1] != '\0'. Now I am thinking that after previously showed patch we maybe need one more fflush(fp); in order to commit append of null-terminator. What do you think?
Hi, sorry for waiting, but we finally managed to get test result without PRIVSEP. Test was tried several times without crash. Any thoughts?
Hi, we managed to detect the root cause. On high icmpv6 RA burst with a lot of route information options, internally dhcpcd builds up huge number of internal environment variables that are also sent to privileged proxy process. It can happen that in ps_root_recvmsgcb() we get environment variables buffer size of 65696 bytes. Seems 65696 is the highest buffer that can be transfered over linux iov socket writev()/read(). And not all data managed to fit. This seems to lead to environment buffer truncation on privileged proxy process receiver side. While on sender side we can see it managed to send cca 84K of bytes, and was null terminated. Not sure could we fix this, by just discarding truncated variable, or we need to introduce splitting of large env buffer into multiple chunks? But with this splitting on sender side I am not sure do we need to have control on privileged proxy process receiver side inside ps_root_recvmsgcb() if we have received full buffer? Please help how to fix this.
Interesting thing is that on dhcpcd-9.4.0 this fault is not visible on sam icmpv6 RA flooding, and it is on dhcpcd-10.2.2. And we have same PRIVSEP mode enabled and ps_sendpsmmsg()/ps_root_recvmsgcb() look very much the same. I am thinking, all those internal variables that are also stored in environment buffer, do they really need to be there? Was that enabled at some point?
We did ipv6nd_env() patch, in a way that we now scans ctx->ra_routers for the newest non-expired rap on this ifp and add only that one (as nd1_*) in the environment buffer. So, we are not placing always all RAs in the environment buffer. Other stuff being added from make_env() is not touched. We don't affect RA parsing , address and route information, that is kept as before. From current testing on our side it looks good, we will send final patch.
Interesting. Maybe it is something within the RA that dhcpcd is trying to translate into env that is causing the crash and not the actual number of RA's in play.
Is the content randomised in any way?
No, RA nd internal variables were not with randomised content. Here is example of output in time of crash when environment buffer reached size of 65696
fdcc nd161_addr1=2001:0:c:461e:569b:72ff:fe20:e8d8/64 fdfd nd162_from=fe80::52d fe12 nd162_acquired=448 fe25 nd162_now=448 fe33 nd162_hoplimit=64 fe45 nd162_flags= fe52 nd162_lifetime=16 fe64 nd162_mtu=1500 fe73 nd162_prefix_information1_length=64 fe97 nd162_prefix_information1_flags=LAH febb nd162_prefix_information1_vltime=4294967295 fee7 nd162_prefix_information1_pltime=4294967295 ff13 nd162_prefix_information1_prefix=2001:0:c:461f:: ff44 nd162_source_address=000c29b95f88 ff66 nd162_addr1=2001:0:c:461f:569b:72ff:fe20:e8d8/64 ff97 nd163_from=fe80::52f ffac nd163_acquired=448 ffbf nd163_now=448 ffcd nd163_hoplimit=64 ffdf nd163_flags= ffec nd163_lifetime=16 fffe nd163_mtu=1500 1000d nd163_prefix_information1_length=64 10031 nd163_prefix_information1_flags=LAH 10055 nd163_prefix_information1_vltime=4294967295 10081 nd163_prefix_information1_pltim
you can see clearly how last variable in this 65696 chunk didn't fit and is truncated as nd163_prefix_information1_pltim with out null termination. This is why script_buftoenv() fails in line 198 assert(*(bufp - 1) == '\0');
and we could see that from sender side in iov socket we have written len cca 84K. So, we didn't see how to handle this as in in icmpv6 RA flooding case this environment buffer would continue to growth. And only thing make sense to allow only adding last RA nd variables to the environment buffer. We have tested this patch solution in ipv6nd_env()
- clock_gettime(CLOCK_MONOTONIC, &now);
- n = 0;
- /* Find newest valid RA */
- TAILQ_FOREACH(rap, ifp->ctx->ra_routers, next) {
-
if (rap->iface != ifp || rap->expired) -
continue; -
if (latest == NULL || -
rap->acquired.tv_sec > latest->acquired.tv_sec || -
(rap->acquired.tv_sec == latest->acquired.tv_sec && -
rap->acquired.tv_nsec > latest->acquired.tv_nsec)) -
latest = rap; - }
- if (latest == NULL)
-
return 0; /* no valid RA */ - rap = latest;
- i = 1;
See full patch attached. We would like to have this included in future dhcpcd versions.
ipv6nd _only_latest_RA_export.txt
We never received reply, to take this patch in github on master. We need to have it as there are some vulnerability testing and we need to be robust on these type of ICMPv6 RA attack.
@Sime-Zupanovic dhcpcd 10.3.0 was just released. Can you tell us whether it fixes it?
It's unlikely to be fixed. If he's using FreeBSD-15 there's a chance though ;)
This and a few others are on my immediate list of things to look at now that we have the new release out.