opnsense
ICMPv6 packets getting sporadically malformed when fragmented over WAN
Important notices
Before you add a new report, we ask you kindly to acknowledge the following:
- [ x] I have read the contributing guide lines at https://github.com/opnsense/core/blob/master/CONTRIBUTING.md
- [ x] I am convinced that my issue is new after having checked both open and closed issues at https://github.com/opnsense/core/issues?q=is%3Aissue
Describe the bug
Setup:
Preface: So in IPv6 intermediate Devices like Firewalls or Routers are not allowed to fragment packets. In the IPv6 specifications it is clearly stated that fragmentation should only occour on the end-devices (Server and Client) and they should avoid fragmentation with PMTUD. If a device however recievs an bigger packet, they will send an ICMPv6 Type 2 Code 0 Message "Too Big" wich will transmit the MTU of the Server and will trigger fragmentation on the Client.
This looks like this:
As you can see my Client is sending a packet with 1400bytes payload and the Servers responds with an "Too Big Message". My Client then fragments the icmp request but for reasons unknown to me, the Server does not respond to my then fragmented request and i get an error Message on my Client.
This however is fine, since opensene handles it correctly.
The BUG
If i ping the host ds.frankfurt.test-ipv6.com [2a01:7e01::f03c:94ff:fed0:4087] and the size of the ping is for example 2500bytes, sometimes it works and sometimes it does not. Actually most of the time it does not work and while I was writting this it suddenly started working. So first I was thinking that this is a clear cut case, but now i am very confused because of the sporadic nature of this bug and I would love some insight from someone that understands it better hopfully.
Here is an example of it Working:
And i can see in the Pcaps taken from WAN and LAN that the packet are the same on WAN and LAN, so ignoring the first ping getting lost, i can see no issue on the opensense side.
Pcap Download: working_wan.zip
However most of the time, and reason for my report, ist that usually the packets on the WAN side, are getting malformed by opensense....
The first fragment, is missing 8 Bytes in the Payload:
LAN - 1448 bytes
WAN - 1440 bytes
This invalidates the icmpv6 checksum but most notably also adds a second ipv6 Fragmentation Header!
Pcap Download: malformed_wan.zip
So either Opensense is not conform to the IPv6 specification and also tries to fragment the packet or there is some kind of error in the handling of this pakets.
To Reproduce
- start capture via Opensense GUI on WAN and LAN:
- run ping ds.frankfurt.test-ipv6.com -l 2500
If you get no response, you will most likely see in the WAN Pcap, that the packets are malformed.
Expected behavior
The packets on the WAN interface should not be malformed by opensene and opensense should not add a second ipv6 fragmentation header. This Ping should work all the time, nomatter the size specified.
Describe alternatives you considered
- It is not a driver / interface issue, since i can reproduce it on igc0 (Intel I225-V) and also on my ixl bond (Intel x-710)
-It is not a upstream ISP Router issue, since if the packet is malformed, it makes sense why it will drop the paket or can not handle the packet if there are multiple fragmentation headers
-Suricata and Traffic Shaping with FQ Codel seem to have no impact on this issue, the behaviour does not change if disabled or enabled
-The sporadic nature of this issue makes it hard to pinpoint.
Additional context
Previous Report: https://github.com/opnsense/core/issues/8744
So this issue was hard for me to pinnpoint and I am no expert when it comes to ipv6, but something is clearly wrong here and I would appreciate it, if someone would test it on their end if it is reproducible. It did cost me alot of time to debug this because everything i looked at made no sense whatsoever or it started to work all of a sudden and then stopped again.
I really hope that someone here can reproduce it and find out why it is happening. I appreciate your time
Environment
Software version used and hardware type if relevant, e.g.: Hardware: Minisofurm MS-01 i9-13900H + 32Gb ddr5 Intel x710 Intel I225-V Bios Version 1.26
ISP Router: Fritzbox 7590 DSL Vectoring with PPPoe
Versions OPNsense 25.1.7_4-amd64 FreeBSD 14.2-RELEASE-p3 OpenSSL 3.0.16
Hello,
I have tested this with my own connection which is quite similiar to yours (IPv6, DTAG - VDSL250, Zyxel modem). I also pinged from a Windows client just like you.
I see the same thing as you, sometimes the ping works, and sometimes it doesn't. And when it doesn't work, I see the same duplicate fragmentation headers in Wireshark.
The question is now, under which circumstances does the ping stop working, and why does it work sometimes. That's the crucial piece of the puzzle to get this further. I have no idea yet, maybe you do?
@Monviech I am unsure,I am just a Network Engineer and no developer. However to me it looks like, that OpenSense or FreeBSD is not conform to the IPv6 Specifications mentioned in RFC 8200
4.5. Fragment Header unlike IPv4, fragmentation in IPv6 is performed only by source nodes, not by routers along a packet's delivery path Source
The RFC mentions Routers, but also mentiones that the Fragmentation is only performend on the source nodes wich a Firewall is not. So I would argue that OpenSense should NEVER fragment IPv6 Traffic that is passing through.
However since this is only happening sporadically, it is probably just a bug and not ignoring the RFC....
What I find interessting, the malformed packets are missing 8 bytes in the payload, 8 bytes is exactly the missing MTU on the ISP Routers WAN Interface because of PPPOE.
So either opensense is trying to Fragment the packet because its PMTU Cache suggests that the MTU is 1492, but this would violate the RFC, but adding another icmp Fragmentation header is wrong in so many ways that I find it hard to find reason in it.
What I would expect to happen ist the following: ping with 2000bytes
- my Client sending one fragmented 1500bytes Payload ICMP packet and one 500bytes fragmented paket to its DFG
- DFG is opensense, Opensense looks in Routing table and checks if MTU is fine with the paket size, wich it is since Opensense has WAN and LAN MTU at 1500 ( If MTU would not be enough, opensense SHOULD answer ICMP Too Big to my Client )
- My ISP Router is recieving the pakets (without manipulation) and sees that the WAN MTU of 1492 is not enough to transfer the first fragment because its payload is 1500bytes and it will send to my client an ICMP Too Big
- My Client recievs the ICMP Too Big Message with the ISP Routers WAN MTU of 1492 and adjusts its PMTU Cache to reflect this change and resends the ICMP Request with the first Fragment having a payload of 1492
- Ping now works, if for example we ping the mtu1280 Host , the process is the same, but the Server will additionaly sent ICMP to Big with MTU 1280 and my client should reflect that in the PMTU Cache.
BTW you can check the PMTU Cache in Windows with
netsh interface ipv6 show destinationcache
and in more details for a ip:
netsh interface ipv6 show destinationcache address=2a01:7e01::f03c:94ff:fed0:4087
@Monviech Ok, so changing the WAN MTU to 1400 has the effect, that the ping "ping ds.frankfurt.test-ipv6.com -l 2000" is working relliably, the first reply is lost downstream but that should be of no concern to us.
The reason as to why the connection works now is interessting....
Changing the MTU on the WAN Interface is actually changing the MTU of the Clients with the Router Advertisments on Tracked Interfaces. You can verify this if you set the WAN MTU to 1400 and run the command "netsh interface ipv6 show subinterface" on the client. You will see that your client has changed its MTU to 1400 aswell for IPv6 only. Leaving the MTU field empty on the WAN, just removes this information from the RA and the Client is stuck with the last setting that has been pushed. Setting WAN MTU to 1500 again also pushes this to the Client aswell.
This behavior is questionable, but if this is communicated on the wiki or GUI, I would not mind it and I could see the benefit of it, I however didnt know that this is a thinq...
But now to the fun part :)
- Setting WAN MTU to 1400
- OverwrittingClient MTU from 1400 -> 1500 with "netsh interface IPv6 set subinterface "Ethernet 3" mtu=1500"
- Start Capturing
- Run ping ds.frankfurt.test-ipv6.com -l 2000 on the client
- CHAOS
On the LAN side, i can verify that the pakets get sent how you expect them to:
On the WAN side, the "BIG" fragment is just gone, not malformed but just outright deleted:
What should happen, is that opensense sends an ICMP too BIG Message to my client to lower the PMTU for this connection to 1400 (since WAN MTU is 1400) so that my Client can resend it.
So in my opinion Opensense is trying to fragment IPv6 packets and if the Payload is bigger then the PMTU / MTU it will either delete the packet or malform it.
Also it seems like opensene has no implementation of ICMPv6 "Too Big" and IPv6 Fragmentation is not IPv6 RFC compliant.
I am unsure how deep this issue goes, is the whole implementation for IPv6 Fragmentation broken or only the icmpv6 fragmentation?
I am also unsure if this is a opensense issue or a FreeBSD issue, the only thing I know is, that I do not have the skill to fix the code and to get it fixed I need your help.
I will also try to get my hands on a Fortigate , Checkpoint or any other Enterprise Firewall to see how they handle this.
Would love to get some input from you
Ok so my final remark for this topic for today. I have now tested how the AVM Fritzbox 7590 does it (My ISP Router). I have connected my PC to the Fritzbox and wanted to confirm what MTU it distributes via RA / SLAC and what happens when i force a MTU that is higher then the WAN MTU.
So the Fritzbox advertises an IPv6 MTU of 1492bytes to the Client, and with this setting fragmentation does not happen. If i run a ping like before it works fine. So as a workaround, if I set the WAN MTU in Opensense to 1492 it works identical like on the ISP Router.
This means that it is actually a configuration error to set the WAN MTU to 1500. But Opensene should if the MTU value is not set and IPv6 is set to DHCPv6 it should learn the correct MTU via the Router Advertisment from the Upstream Router. This is currently not the Case and if it is empty it assumes 1500bytes. (Maybe a Feature Request?!)
However we still have an Bug here, Opensene needs to stop messing with IPv6 Fragments if they exceed the MTU of the Interface and fragmentation is need, opensene need to inform the client with ICMPv6 Typ 2 "Too Big" Messages that fragmentation is necessary and send the correct MTU.
The Fritzbox handels it well:
- Fritzbox advertises MTU of 1492 via RA / SLAC to the PC
- I overwritte the MTU on the Client with "netsh interface IPv6 set subinterface "Ethernet 3" mtu=1500"
- I ping an IPv6 host who allows fragmentation , for example : ping blog.fefe.de -l 2000 or ping ds.frankfurt.test-ipv6.com -l 2000
The Result must look like this:
PCAP Download: fritzbox_correct_ipv6.zip
As you can see the fritzbox implements IPv6 correctly and sents an ICMPv6 Typ 2 Message to my Client with MTU hint of 1492 so that my Client can resend the packet with the correct MTU.
to summarise
- OpenSense mustn't fragment IPv6 Traffic that is passing through the Firewall (RFC 8200)
- Opensene must implement ICMPv6 Typ 2 to inform clients, when a Frame has been recieved that is bigger then the MTU of the destination Interface so that the Client knows that it needs to resend the paket with the MTU value transmitted via ICMPv6 Type 2 (RFC 8200)
- [Feature Request]Opensene should honor the IPv6 MTU that upstream routers transmitt via Router Advertisments when the Interface Configuration is set to DHCPv6 and no MTU has been specified for the Interface
If you can fix and implement these 3 points, it will work how it is intended
what effect has disable scrubbing on your issue? (Firewall: Settings: Normalization / Disable interface scrub)
@AdSchellevis Disabling FW Normalization like you mentioned, changes nothing in the IPv6 flow:
ISP Router LAN MTU= 1500 ISP Router WAN MTU = 1492 Opensene WAN MTU = 1500 --> RA to Client 1500 MTU Opensene LAN MTU = 1500
Client IPv6 MTU = 1500
Opensene LAN side:
Opensene WAN Side:
The packets on WAN are still missing 8 bytes and opensense still adds a second fragmentation header.
For IPv4 this makes everything wierd.... So if interface scrub is disabled the paket flow is fine, but opensene does not transmitt the recieved ICMP Reply from WAN to LAN and my client gets no answer.
Opensene LAN side:
Opensene WAN side:
This reply is just deleted on WAN, it makes no sense to me. Sure the ISP Gateway seems to send 3 Fragments instead of 2 but the sizes of these fragments do not exceed MTU on any interface.
Enabling Normalisatzion again fixes this, and the flow is working fine
EDIT
Ok so it does make sense to me, since disabling Normalisiatzion disables ICMP statefullness and I get FW Rule blocks:
For ipv4 I'm quite sure it works as designed, for ipv6 we might be chasing ghosts here, although I'm not sure. If you want to exclude the firewall in your tests, for ipv6, you should also be able to disable pf (pfct -d and pfctl -e to enable) to see if fragmentation still happens. To some degree I expect a logical explanation, but surprises exist.
For ipv4 I'm quite sure it works as designed, for ipv6 we might be chasing ghosts here, although I'm not sure. If you want to exclude the firewall in your tests, for ipv6, you should also be able to disable pf (
pfct -dandpfctl -eto enable) to see if fragmentation still happens. To some degree I expect a logical explanation, but surprises exist.
I am with you that it is correct in IPv4
But I am certain that Opensense (or FreeBSD) is fragmenting IPv6 packets and that is just plain wrong behaviour. In IPv6 the firewall / Router should NEVER mess with the packet. Instead it should send these ICMPv6 Typ 2 Messages to inform the Client.
I will test the commands (pfct) later today
@AdSchellevis i can confirm that the issue is not happening when i disable the firewall with pfctl -d
I have also set the MTU on WAN to 1500 again
Without pf enabled the LAN Interface IPv6 sents ICMPv6 Typ 2 "Too Big" as seen in the following picture:
On the WAN Interface you will not see the first fragment, this is most likly the case, because the Firewall Device knows the Path MTU already, but i do not know how to check the PMTU cache in FreeBSD.
I would need to test this more precisly, what happens if there are no Gateway Monitoring enabled, what happens when i reboot the system and clear the caches etc...
But this works, the goal is that the client is informed that the Fragments are too big and this happens here, it does not matter if opensene or the isp router inform the client.
So does that mean it is a opensense issue and not a freebsd one?
well, if the behavior is different with pf disabled, you might try to create a very simple ruleset (pass all) to see if there's something in our default ruleset (although I don't expect that's the case).
Well i added an ipv4+ipv6 rule with any on both LAN and WAN, but it didnt changed the outcome.
just to be sure, I mean a pf.conf file with only pass all in it to exclude any defaults from our end (loaded via pfctl -f /path/to/my/file)
just to be sure, I mean a pf.conf file with only
pass allin it to exclude any defaults from our end (loaded viapfctl -f /path/to/my/file)
Sure can do that, but would you be so kind to tell me the path of my current config so that i can back it up first? It is still my live Home Network and not a lab setup
Your current configuration exists in /conf/config.xml
Loading a file into PF manually won't change the main configuration file of the firewall, though.
Any "Apply" in the Firewall/Rules pages, restarting the packet filter in the GUI under services, or a reboot restores the correct ruleset.
@Monviech do you have a file to test with? Because if I create a simple file "test.xml" with "pass all" i just get a syntax error
I tried some different xml syntax like in the current config but that also didnt work, sorry I never edited pf or its configs
I think you misunderstood. I thought you wanted to backup your current firewall configuration.
The current generated pf rules are stored in /tmp/rules.debug
Thats the format your file needs, but only with a pass all rule. Just create an any any rule in the GUI in Floating, give it an easy to find description, find it in rules.debug, and copy it into your own rules.test file, and then load it with pfctl.
To see the current loaded rules, check pfctl -s rules.
pf is essentially a kernel module in freebsd that is the packet filter. Here is an example cheat sheet with some commands to control it:
https://bsdwatch.net/articles/pfctl-cheatsheet
I'll help tomorrow if you are stuck.
On a different note, does opensense implement ipfw?
Because quite some time ago, there was this report https://reviews.freebsd.org/D10533 and the linked bugreport https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=170604 reads alot like the problem i am seeing here.
What i find a bit troubling, is that the patch of Andrey V. Elsukov looks to be implemented and after what he described, the implementation actually just fragments ? But to be honest both the patch and the discussion in the bug report are depressing
Edit:
oh noooo, maybe its ipfw....
Without changing my rules, and just loading my test.xml with the following entrys: pass quick inet from {any} to {any} keep state label "6563996381efaf45b6258349f0bf9640" # TEST RULE pass quick inet6 from {any} to {any} keep state label "6563996381efaf45b6258349f0bf9640" # TEST RULE
It didnt seem to change anything
I think i may have found somthing that is current: https://reviews.freebsd.org/rGd1f4b9ffabbcc2c29ab83435bd73b0670818bbd1
This is a commit from april of 2025 in the pf_normalisation about icmp "Too BiG" not actually beeing sent when "route-to" is used
And this would make sense, since this Problem is only happening on the WAN, i cant replicate it in the LAN.
I use "Gateways" do these implement route-to?
The route-to rules are created when in "Interface -> WAN (e.g.)" the "IPv4" or "IPv6" gateway rules dropdown is populated with a gateway (when the configuration is static). If it's DHCP, they are automatically created.
You should be able to see route-to rules in your standard pf ruleset if you do something like:
cat rules.debug | grep -i route-to
Since I can recreate the same issue in my setup I will give you a pf file to test later after I verified it on my end.
The route-to rules are created when in "Interface -> WAN (e.g.)" the "IPv4" or "IPv6" gateway rules dropdown is populated with a gateway (when the configuration is static). If it's DHCP, they are automatically created.
You should be able to see route-to rules in your standard pf ruleset if you do something like:
cat rules.debug | grep -i route-toSince I can recreate the same issue in my setup I will give you a pf file to test later after I verified it on my end.
Looks like we a on the right track
only one route-to Rule that is IPv6 only and on that interface the issue is happening
Now how to verify this? Can we work around this route-to ?
You could try:
cp /tmp/rules.debug /tmp/rules.debug.test
vi /tmp/rules.debug.test
- Inside vi:
/route-to
Press Enter
Use Up down keys to select the correct route to line
Press dd
Press ESC
Press :wq
pfctl -f /tmp/rules.debug.test
What we essentially did here is remove just the IPv6 route-to line selectively from the ruleset, and loaded that file.
Check with
pfctl -s rules | grep -i route-to
Don't press anything in the WebGUI like applying interface settings or applying firewall rules, otherwise the state reverts to before this.
@Monviech i will test your commands later. I can however tell you that i have removed it in a different way, so this rule (in my rule set) ist depended on the following Firewall Setting:
After disabling this, there was no route-to rule anymore, but IPv6 connectivity was broken. I then added a static ipv6 route , confirmed that it does not trigger a Route-to rule to be created and tested again.
However my IPv6 connectivity was still broken, and it didnt matter what i added, no route or fw rule fixed it. Even enabling this setting again didnt help... I even clear all states....
So i dont know what caused this but only a reboot of opensene re enabled ipv6
As my manual way would have the same result as this Webgui setting, you can skip this.
I have to test this myself but I didnt have time yet.
1.add static ipv4 route 2. add static ipv6 route like shown under route status 3. enable "disable force gateway" 4. verify with "cat /tmp/rules.debug | grep -i "route-to"" that we do not have a entry 5. packet capture
So with that ipv6 connectivity was not broken this time, but it didnt change the outcome. This time the BIG Packet is just getting dropped on the WAN Interface , so LAN->WAN has packetloss My client has also 1500mtu in its destination cache because of SLAC advertisment with 1500 because of the WAN MTU
So my WAN MTU is 1500 and my LAN MTU is 1500, why is the big packet just dropped? It would make sense if we have some kind of MTU / PMTU Cache like in windows, because opensene should know at this point that the ipv6 MTU upstream is only 1492 bytes
But this is quite problematic:
- In its default behavior with "Gateway" and "Policy Based Routing", pf or ipfw tries to fragment the paket with the PMTU he has in cache to 1492 bytes, but this fragmentation function seems to be broken. It removes 8 bytes from the payload but adds a second IPv6 Fragmentation Header wich invalidates the paket. Also no ICMPv6 Typ 2 will be sent to the Client.
Result: The Upstream Router can not understand this malformed paket, also checksum is broken and the PMTU to the Client is broken. Fix: pf or ipfw should never fragment traffic because this is a violation of RFC-8200, also it needs to inform the client that the MTU for this path is 1492. If PF or ipfw do not know the PMTU for that path and work with Interface MTU of 1500 it should transmit the packet upstream, and the upstream router will answer with ICMPv6 Type2 "Too Big". This Message needs to be sent downstream to the Client and can optionaly be saved into pf/ipfw pmtu cache so that it can sent ICMPv6 Type 2 for future packets if necaessary
- If we disable "Policy Based Routing" or the packet size is bigger then the MTU of the egress interface (WAN) like in my test from this comment https://github.com/opnsense/src/issues/254 or the PMTU Cache shows a smaller PMTU then the packet,
Then pf/Ipfw drops the big packets, wich is actually correct, but it needs to sent ICMPv6 Type 2 as well and this seems to be not implemented currently. FreeBSD does this correct, so in the Operating system it is working fine, and we can verify this with disabling pf.
So where does this leave us? I dont know how freebsd / pf and ipfw are stacked in opensense, disabling pf seems to fix it, so i tend to say "PF has no implementation of ICMPv6 Type 2" atleast right now. Also the current "reass" implementation most likly in ipfw is broken because it looks to me like ipfw is responsibel for shaping, PBR etc...but ipfw is not RFC 8200 compliant.
i cannot fathom why in 2017 they decieded to implement it broken and why this is still a thing in 2025. The only silverlining here for me is, that in TCP this is not an issue because of TCP MSS and mss clamping.
UDP is most of the time small and ICMP well, who tests fragmentation anyways... I am a bit unsure about DTLS connections, because some VPN Clients use it and we have no TCP mss to safe us.
I dug a little into https://cgit.freebsd.org/src/commit/?id=d1f4b9ffabbc which lead to https://cgit.freebsd.org/src/commit/?id=56b7685ae32 which is all features not found in FreeBSD 14 branch meaning no FreeBSD before 15.0 will support this. https://www.freebsd.org/releases/15.0R/schedule/ says ETA for 15.0 is December 2025. We always skip initial releases so won't update before 15.1 which could easily be 26.7 or 27.1 to give a sense of time frame here.
If FreeBSD will not backport to 14 we will not try to attempt it either before it lands in a real world release that is 15.0.
Cheers, Franco
@fichtner Any relation to https://github.com/opnsense/core/issues/7342 ? A couple posters there observe ICMPv6 fragmentation, but it's with regard to shaping with ipfw.
I just found this bug. Just to add some data points to this, because I have inconclusive results:
-
When trying from my home connection via PPPoE/VLAN with an optimized 1500 byte MTU via "mini jumbo frames", I see pings working up to a size of 1452 bytes, with large packets, no dice, but consistently. This is from a linux client.
-
I see the same from a connection at Hetzner with static IP configuration and 1500 byte MTU, apart from the fact that I see an answer from the router, which happens to be my own Proxmox server:
From 2a01:4f9:3080:f6b4::180 icmp_seq=1 Packet too big: mtu=1500 -
When I do the same test from a Windows 11 client with MTU=1500 on my home network, I get essentially the same results when I request "dont fragment (-f)", but I can see that fragmentation only works for IPv4, not for IPv6, when I try longer packets.
-
For Win 11 under Proxmox on a Hetzner host it works up to 1452 bytes and will give an error ("Allgemeiner Fehler") up to a value of 1456, then just timeouts. For IPv6, "dont fragment" cannot be requested, BTW.
