[BUG] WAN interface periodically losing DHCP lease

[0xC0FFEEEE]

Important notices

Before you add a new report, we ask you kindly to acknowledge the following:

• [x] I have read the contributing guide lines at https://github.com/opnsense/core/blob/master/CONTRIBUTING.md
• [x] I am convinced that my issue is new after having checked both open and closed issues at https://github.com/opnsense/core/issues?q=is%3Aissue

Describe the bug

Similar to other users, my WAN interface will periodically drop its lease (at least once a day) and won't recover until I reconfigure the wan interface or issue dhclient igb0. This has plagued me for around a year using both pfsense and opnsense.

I've tried the following:

• Setting supersede dhcp-lease-time 21600; this delayed the issue from re-occurring, but would still happen roughly once every 5 days.
• Setting a daily cron task to re-configure the wan, which has effectively nullified the improvement from the change above.
• Have a packet capture job running to capture DHCP traffic, this showed no packets when the issue last ocurred (may be a red herring).

There's nothing in the logs when this occurs, only the result of the cron (04:05) or manual intervention after the issue had occurred (10:46):

<13>1 2024-05-22T04:05:04+00:00 OPNsense.localdomain opnsense 80832 - [meta sequenceId="33"] /usr/local/etc/rc.newwanip: plugins_configure vpn (execute task : openvpn_configure_do(,wan))
<13>1 2024-05-22T04:05:04+00:00 OPNsense.localdomain opnsense 80832 - [meta sequenceId="31"] /usr/local/etc/rc.newwanip: plugins_configure vpn (,wan)
<13>1 2024-05-22T04:05:04+00:00 OPNsense.localdomain opnsense 80832 - [meta sequenceId="32"] /usr/local/etc/rc.newwanip: plugins_configure vpn (execute task : ipsec_configure_do(,wan))
<13>1 2024-05-22T04:05:04+00:00 OPNsense.localdomain opnsense 80832 - [meta sequenceId="33"] /usr/local/etc/rc.newwanip: plugins_configure vpn (execute task : openvpn_configure_do(,wan))
<13>1 2024-05-22T04:05:04+00:00 OPNsense.localdomain opnsense 80832 - [meta sequenceId="34"] /usr/local/etc/rc.newwanip: Resyncing OpenVPN instances for interface WAN.
<13>1 2024-05-22T04:05:04+00:00 OPNsense.localdomain opnsense 80832 - [meta sequenceId="35"] /usr/local/etc/rc.newwanip: plugins_configure vpn (execute task : wireguard_configure_do(,wan))
<13>1 2024-05-22T04:05:04+00:00 OPNsense.localdomain opnsense 80832 - [meta sequenceId="36"] /usr/local/etc/rc.newwanip: plugins_configure newwanip (,wan)
<13>1 2024-05-22T04:05:04+00:00 OPNsense.localdomain opnsense 80832 - [meta sequenceId="37"] /usr/local/etc/rc.newwanip: plugins_configure newwanip (execute task : dnsmasq_configure_do())
<13>1 2024-05-22T04:05:04+00:00 OPNsense.localdomain opnsense 80832 - [meta sequenceId="38"] /usr/local/etc/rc.newwanip: plugins_configure newwanip (execute task : miniupnpd_configure_do())
<13>1 2024-05-22T04:05:04+00:00 OPNsense.localdomain opnsense 80832 - [meta sequenceId="39"] /usr/local/etc/rc.newwanip: plugins_configure newwanip (execute task : ntpd_configure_do())
<13>1 2024-05-22T04:05:04+00:00 OPNsense.localdomain opnsense 80832 - [meta sequenceId="40"] /usr/local/etc/rc.newwanip: plugins_configure newwanip (execute task : opendns_configure_do())
<13>1 2024-05-22T04:05:04+00:00 OPNsense.localdomain opnsense 80832 - [meta sequenceId="41"] /usr/local/etc/rc.newwanip: plugins_configure newwanip (execute task : openssh_configure_do(,wan))
<13>1 2024-05-22T04:05:04+00:00 OPNsense.localdomain opnsense 80832 - [meta sequenceId="42"] /usr/local/etc/rc.newwanip: plugins_configure newwanip (execute task : unbound_configure_do(,wan))
<13>1 2024-05-22T04:05:13+00:00 OPNsense.localdomain opnsense 80832 - [meta sequenceId="43"] /usr/local/etc/rc.newwanip: plugins_configure newwanip (execute task : vxlan_configure_do())
<13>1 2024-05-22T04:05:13+00:00 OPNsense.localdomain opnsense 80832 - [meta sequenceId="44"] /usr/local/etc/rc.newwanip: plugins_configure newwanip (execute task : webgui_configure_do(,wan))
<13>1 2024-05-22T04:05:13+00:00 OPNsense.localdomain opnsense 80832 - [meta sequenceId="45"] /usr/local/etc/rc.newwanip: plugins_configure newwanip (execute task : wireguard_sync(,wan))
<27>1 2024-05-22T10:46:58+00:00 OPNsense.localdomain dhclient 56367 - [meta sequenceId="1"] My address (<redacted>) was deleted, dhclient exiting
<13>1 2024-05-22T10:46:58+00:00 OPNsense.localdomain dhclient 62228 - [meta sequenceId="2"] New IP Address (igb0): <redacted>
<13>1 2024-05-22T10:46:58+00:00 OPNsense.localdomain dhclient 64212 - [meta sequenceId="3"] dhclient-script: Reason FAIL on igb0 executing
<27>1 2024-05-22T10:46:58+00:00 OPNsense.localdomain dhclient 56367 - [meta sequenceId="4"] connection closed
<26>1 2024-05-22T10:46:58+00:00 OPNsense.localdomain dhclient 56367 - [meta sequenceId="5"] exiting.
<13>1 2024-05-22T10:46:58+00:00 OPNsense.localdomain dhclient 66008 - [meta sequenceId="6"] New Subnet Mask (igb0): 255.255.255.0
<13>1 2024-05-22T10:46:58+00:00 OPNsense.localdomain dhclient 68484 - [meta sequenceId="7"] New Broadcast Address (igb0): <redacted>
<13>1 2024-05-22T10:46:58+00:00 OPNsense.localdomain dhclient 69948 - [meta sequenceId="8"] New Routers (igb0): <redacted>
<13>1 2024-05-22T10:47:19+00:00 OPNsense.localdomain kernel - - [meta sequenceId="9"] <6>ovpnc3: link state changed to DOWN
<13>1 2024-05-22T10:47:20+00:00 OPNsense.localdomain kernel - - [meta sequenceId="10"] <6>ovpnc3: link state changed to UP
<13>1 2024-05-22T10:47:21+00:00 OPNsense.localdomain opnsense 28568 - [meta sequenceId="11"] /usr/local/etc/rc.newwanip: IP renewal starting (new: 10.96.0.29, old: , interface: opt3, device: ovpnc3, force: yes)
<13>1 2024-05-22T10:47:21+00:00 OPNsense.localdomain opnsense 28568 - [meta sequenceId="12"] /usr/local/etc/rc.newwanip: ROUTING: entering configure using 'opt3'
<13>1 2024-05-22T10:47:22+00:00 OPNsense.localdomain opnsense 28568 - [meta sequenceId="13"] /usr/local/etc/rc.newwanip: plugins_configure monitor (,PROTONVPN_VPNV4)
<13>1 2024-05-22T10:47:22+00:00 OPNsense.localdomain opnsense 28568 - [meta sequenceId="14"] /usr/local/etc/rc.newwanip: plugins_configure monitor (execute task : dpinger_configure_do(,PROTONVPN_VPNV4))
<13>1 2024-05-22T10:47:24+00:00 OPNsense.localdomain opnsense 28568 - [meta sequenceId="15"] /usr/local/etc/rc.newwanip: plugins_configure vpn (,opt3)
<13>1 2024-05-22T10:47:24+00:00 OPNsense.localdomain opnsense 28568 - [meta sequenceId="16"] /usr/local/etc/rc.newwanip: plugins_configure vpn (execute task : ipsec_configure_do(,opt3))
<13>1 2024-05-22T10:47:24+00:00 OPNsense.localdomain opnsense 28568 - [meta sequenceId="17"] /usr/local/etc/rc.newwanip: plugins_configure vpn (execute task : openvpn_configure_do(,opt3))
<13>1 2024-05-22T10:47:24+00:00 OPNsense.localdomain opnsense 28568 - [meta sequenceId="18"] /usr/local/etc/rc.newwanip: Resyncing OpenVPN instances for interface ProtonVPN.
<13>1 2024-05-22T10:47:24+00:00 OPNsense.localdomain opnsense 28568 - [meta sequenceId="19"] /usr/local/etc/rc.newwanip: plugins_configure vpn (execute task : wireguard_configure_do(,opt3))
<13>1 2024-05-22T10:47:24+00:00 OPNsense.localdomain opnsense 28568 - [meta sequenceId="20"] /usr/local/etc/rc.newwanip: plugins_configure newwanip (,opt3)
<13>1 2024-05-22T10:47:24+00:00 OPNsense.localdomain opnsense 28568 - [meta sequenceId="21"] /usr/local/etc/rc.newwanip: plugins_configure newwanip (execute task : dnsmasq_configure_do())
<13>1 2024-05-22T10:47:24+00:00 OPNsense.localdomain opnsense 28568 - [meta sequenceId="22"] /usr/local/etc/rc.newwanip: plugins_configure newwanip (execute task : miniupnpd_configure_do())
<13>1 2024-05-22T10:47:24+00:00 OPNsense.localdomain opnsense 28568 - [meta sequenceId="23"] /usr/local/etc/rc.newwanip: plugins_configure newwanip (execute task : ntpd_configure_do())
<13>1 2024-05-22T10:47:24+00:00 OPNsense.localdomain opnsense 28568 - [meta sequenceId="24"] /usr/local/etc/rc.newwanip: plugins_configure newwanip (execute task : opendns_configure_do())
<13>1 2024-05-22T10:47:24+00:00 OPNsense.localdomain opnsense 28568 - [meta sequenceId="25"] /usr/local/etc/rc.newwanip: plugins_configure newwanip (execute task : openssh_configure_do(,opt3))
<13>1 2024-05-22T10:47:24+00:00 OPNsense.localdomain opnsense 28568 - [meta sequenceId="26"] /usr/local/etc/rc.newwanip: plugins_configure newwanip (execute task : unbound_configure_do(,opt3))
<13>1 2024-05-22T10:47:24+00:00 OPNsense.localdomain opnsense 28568 - [meta sequenceId="27"] /usr/local/etc/rc.newwanip: plugins_configure newwanip (execute task : vxlan_configure_do())
<13>1 2024-05-22T10:47:24+00:00 OPNsense.localdomain opnsense 28568 - [meta sequenceId="28"] /usr/local/etc/rc.newwanip: plugins_configure newwanip (execute task : webgui_configure_do(,opt3))
<13>1 2024-05-22T10:47:24+00:00 OPNsense.localdomain opnsense 28568 - [meta sequenceId="29"] /usr/local/etc/rc.newwanip: plugins_configure newwanip (execute task : wireguard_sync(,opt3))

I suspect that dhclient is failing/exiting ungracefully, as in the case above, I issued dhclient igb0 to restore connectivity:

root@OPNsense:~ # dhclient igb0
DHCPREQUEST on igb0 to 255.255.255.255 port 67
DHCPACK from <redacted>
bound to <redacted> -- renewal in 43200 seconds.

Running the same command after the fact results in the following:

root@OPNsense:~ # dhclient igb0
dhclient already running, pid: 95681.
exiting.

Tip: to validate your setup was working with the previous version, use opnsense-revert (https://docs.opnsense.org/manual/opnsense_tools.html#opnsense-revert)

To Reproduce

Steps to reproduce the behavior:

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Expected behavior

dhclient renews lease as required.

Describe alternatives you considered

A clear and concise description of any alternative solutions or workaround you considered.

Screenshots

If applicable, add screenshots to help explain your problem.

Relevant log files

If applicable, information from log files supporting your claim.

Additional context

Add any other context about the problem here.

Environment

Software version used and hardware type if relevant, e.g.:

OPNsense 24.1.7_4 (amd64). Intel® Celeron™ J3160 1.60GHz Quad Core Network Intel® I210