IPv6 MLD listener reports wrongly go over the PPPoE interface

[martijncoenen]

Important notices

Before you add a new report, we ask you kindly to acknowledge the following:

• [X] I have read the contributing guide lines at https://github.com/opnsense/core/blob/master/CONTRIBUTING.md
• [X] I am convinced that my issue is new after having checked both open and closed issues at https://github.com/opnsense/core/issues?q=is%3Aissue

Describe the bug

I have had this issue for a really long time, and originally created this issue for it. I tried to reopen it, but I can't. I am opening a new issue because I have more information now on what the root cause is. TLDR, IPv6 MLD packets erroneously get sent out on the pppoe interface, whereas they should have been sent out on the local LAN interface. I had kind of given up on it and considered it user error, until another user reported running into the same issue.

So, I took a journey through the FreeBSD kernel, pf, the mpd PPP daemon, and this is what I found:

The kernel correctly sends out the MLD report on the LAN interface. However, OPNSense has a default firewall rule present that forces packets that have a source address of a gateway, to also go out over said gateway. By itself this seems totally fine - why would these MLD packets on the LAN have the same source address as the pppoe interface? Well, it turns out that the mpd5 ppp daemon just picks a random interface to determine its own link-local address, and in my case that happened to be the LAN interface. So:

root@opnsense:/tmp # ifconfig vtnet1_vlan100 | grep fe80
        inet6 fe80::9ca3:3dff:fea4:9380%vtnet1_vlan100 prefixlen 64 scopeid 0x12
root@opnsense:/tmp # ifconfig pppoe0 | grep fe80
        inet6 fe80::9ca3:3dff:fea4:9380%pppoe0 prefixlen 64 scopeid 0x18

Both pppoe and vtnet1_vlan100 (the LAN interface) have the same IPv6 link-local address. By itself, that seems fine, too - these are separate links and therefore having the same link-local address doesn't really matter. But, in combination with the firewall rule I mentioned earlier, these MLD packets, which originate from fe80::9ca3:3dff:fea4:9380, now wrongly get sent out over the pppoe0 interface instead.

I found that enabling 'Firewall -> Settings -> Advanced -> Disable automatic rules which force local services to use the assigned interface gateway' removes these rules, and indeed that fixes the issue, both for me and the other user that reported it.

I'm not a network expert so I'm not sure what actually the real proper fix is. My intuition is that it is fine for OPNSense to give out the same LLA to two different interfaces, but if it does, it should not add firewall rules that act on that ambiguity. Maybe the 'force packet with source address of gateway to go out over said gateway' should not do that for link-local addresses.

To Reproduce

I think you need a few things to reproduce this:

• The WAN must be using PPPoE to connect (to get an identical link-local address on the PPP interface)
• You must be lucky (?) that the mpd PPP stack enumerates your LAN adapter first, since that is what it will use the link-local address from
• You must have a network where MLD queries are sent out, and where OPNSense is configured such that it will respond to them. I think that means OPNSense has to be configured as IPv6 such that it will act as on IPv6 router, which means OPNSense will at least respond to MLD queries with the ff02::2 address.
• 'Disable automatic rules which force local services...' must be disabled

When these preconditions are met, you should be able to observe MLD responses go out on the PPP interface.

Expected behavior

MLD packets destined for a particular interface should also go out on that interface.

Describe alternatives you considered

N/A

Screenshots

N/A

Relevant log files

N/A

Additional context

N/A

Environment

OPNsense 25.1.7_4-amd64, virtualized in Proxmox Ryzen 3700X virtio network adapters (backed by proxmox bridge)

[AdSchellevis]

vtnet1_vlan100 is not the interface pppoe is bound too? That might be problematic to fix with automatic rules as these aren't (and won't be) fine grained. We are considering adding a couple of questions in the wizard to make people more aware of multiwan scenario's, but nothing concrete yet.

What is the output of grep route-to /tmp/rules.debug on your end by the way?

[martijncoenen]

vtnet1_vlan100 is not the interface pppoe is bound too

No, in my case pppoe is bound to vtnet0_vlan6 - an entirely different interface for the WAN. You could say ideally the PPP daemon should pick the link-local address of the parent interface (eg vtnet0 or vtnet0_vlan6), but it appears to just pick the first interface in the list of all interfaces, which is pretty random.

What is the output of grep route-to /tmp/rules.debug on your end by the way?

With the disable automatic rules which force local services... setting disabled (the default), it is:

pass out route-to ( ovpnc1 10.100.0.1 ) from {(ovpnc1)} to {!(ovpnc1:network)} keep state allow-opts label "d4cb16cc171804cddd04061e6d2145e7" # let out anything from firewall host itself (force gw)
pass out route-to ( pppoe0 195.190.xxx.xxx ) from {(pppoe0)} to {!(pppoe0:network)} keep state allow-opts label "badf9fd7b03523686df3cda925091a44" # let out anything from firewall host itself (force gw)
pass out route-to ( pppoe0 fe80::627e:cdff:fe29:c73b ) from {(pppoe0)} to {!(pppoe0:network)} keep state allow-opts label "b8396040dd4b11ecf36f9ad451d44831" # let out anything from firewall host itself (force gw)
pass out route-to ( ovpnc1 10.100.0.1 ) inet from {(ovpnc1)} to {!(ovpnc1:network)} keep state allow-opts label "5168bd9a3df4aad097c97c20268a8549" # Route to NordVPN
pass in quick on vtnet1_vlan100 route-to ( ovpnc1 10.100.0.1 ) inet from $Routed_to_NordVPN to !$LocalNetworks keep state label "db521f5aeb49d8f0e04f79709b1199a5" # Route to NordVPN
pass in log quick on vlan02 route-to ( ovpnc1 10.100.0.1 ) inet from $Routed_to_NordVPN to !$LocalNetworks keep state label "88787171aa90633063b9a24a360f8087"

I believe the third rule is the problematic one, because it forces the MLD packets out over pppoe0.

If I enable the 'disable automatic rules...` setting (which fixes the issue), it is:

pass out route-to ( ovpnc1 10.100.0.1 ) inet from {(ovpnc1)} to {!(ovpnc1:network)} keep state allow-opts label "5168bd9a3df4aad097c97c20268a8549" # Route to NordVPN
pass in quick on vtnet1_vlan100 route-to ( ovpnc1 10.100.0.1 ) inet from $Routed_to_NordVPN to !$LocalNetworks keep state label "db521f5aeb49d8f0e04f79709b1199a5" # Route to NordVPN
pass in log quick on vlan02 route-to ( ovpnc1 10.100.0.1 ) inet from $Routed_to_NordVPN to !$LocalNetworks keep state label "88787171aa90633063b9a24a360f8087"

[AdSchellevis]

does pppoe0 have a GUA assigned to it?

[martijncoenen]

does pppoe0 have a GUA assigned to it?

No, just a link-local:

root@opnsense:~ # ifconfig pppoe0
pppoe0: flags=10088d1<UP,POINTOPOINT,RUNNING,NOARP,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1492
        options=0
        inet 77.174.xxx.xxx --> 195.190.xxx.xxx netmask 0xffffffff
        inet6 fe80::9ca3:3dff:fea4:9380%pppoe0 prefixlen 64 scopeid 0x18
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

I guess this is because I'm requesting only a prefix for IPv6.

[martijncoenen]

Also paging @T-X, who had the exact same issue (and the same workaround fixed it), but maybe there are configuration differences.

[AdSchellevis]

@martijncoenen can you try https://github.com/opnsense/core/commit/5018fc8d9df3bbb2a3f8fb195e6b47dc4b275521?

opnsense-patch 5018fc8

[martijncoenen]

@martijncoenen can you try 5018fc8?

opnsense-patch 5018fc8

Thanks, it looks like that also deals with the issue, the offending rule is no longer generated. I do wonder: for configurations where the PPPoE interface does have a GUA, wouldn't that still run into the same problem - eg the generated rule would still match packets that have the LLA as a source?

[AdSchellevis]

We haven't seen issues with it, at first I was thinking the same, but this might be pf specific (the to alias is empty when only a link local address is specified). I do doubt the usefulness of these rules for ipv6 a bit in most cases, but don't want to change too much here at the moment. Manual rules always work and can be fine grained when needed.

[martijncoenen]

We haven't seen issues with it, at first I was thinking the same, but this might be pf specific (the to alias is empty when only a link local address is specified). I do doubt the usefulness of these rules for ipv6 a bit in most cases, but don't want to change too much here at the moment. Manual rules always work and can be fine grained when needed.

Ah, if the alias becomes more specific once there is a GUA that makes sense. Understand you want to be careful around changes here. The questions in the wizard do seem useful, IIUC these rules are mostly useful in a multi-WAN setup. Thanks again for the quick fix here!

[AdSchellevis]

I'll keep the ticket open for a while to discuss this further internally, for now this seems to be a safe (and sensible) fix

[fichtner]

Looks like it's working :)