OpenVPN
Error "read UDPv4 [CMSG=8|EMSGSIZE Path-MTU=1492]: Message too large" only with packet sizes of 1441 to 1448
I have found that inside of an Openvpn tunnel if I send ICMP packets ranging in size from 1441 to 1448 bytes, they fail to go through the vpn. Any other size works. When packets get lost, I have an error that says "read UDPv4 [CMSG=8|EMSGSIZE Path-MTU=1492]: Message too large (fd=7,code=90)"
My setup is as follows:
Openvpn 2.6.14 on Openwrt, set up as a server. The config is very simple and there are no settings regarding tunnel MTU, link MTU, etc. Proto is UDP.
The external link is PPPOE so the MTU of the link is 1492 (tested it with ping -M do -s 1492 <external remote ip>). Please note that PPPOE is not terminated on the Openvpn server so its egress interface MTU is 1500, only the PATH MTU is 1492.
Openvpn 2.6.12 on Linux, on the client side. The config is basically the same as the server, no settings regarding MTU.
In both logs (server and client ) the only information I see about MTU settings are there:
Control Channel MTU parms [ mss_fix:0 max_frag:0 tun_mtu:1250 tun_max_mtu:0 headroom:126 payload:1600 tailroom:126 ET:0 ]
Data Channel MTU parms [ mss_fix:0 max_frag:0 tun_mtu:1500 tun_max_mtu:1600 headroom:136 payload:1768 tailroom:562 ET:0 ]
And then after the connection is established, I see this new information in the log:
Data Channel MTU parms [ mss_fix:1400 max_frag:0 tun_mtu:1500 tun_max_mtu:1600 headroom:136 payload:1768 tailroom:562 ET:0 ]
Once the VPN is connected, if I ping INSIDE of the VPN, I see this behavior:
ping -s 1412 192.168.4.10 - WORKS
ping -s 1413 192.168.4.10 - FAILS, no answer. (I get the message in the log ONLY ON SERVER SIDE)
ping -s 1414 and up to 1420, FAILS, as before.
ping -s 1421 192.168.4.10 - WORKS
every other size over 1421 works, too.
Please note that the actual size of the packet generated by these ping commands is bigger than the "-s" size I stated. The failing packet sizes are 1441 to 1448.
I have no idea why this happens. I have tried with Openvpn 2.5 and the issue is the same.
Please note that the error in the log appears only server-side and that the server is OpenWrt (on a AMD64 cpu).
Basically sending large UDP packets on your networks fails. There is not really much that OpenVPN can do about that. The OS should received the ICMP packets and adjust the UDP fragments it is sending but apparently that is not happening in your case. But if the packets get larger again it is splitting them before into 2 fragments.If you had a larger tunnel mtu you would see another point where it breaks again when the two fragments get to large.
You can use --fragment in this situation or rely on mssfix for TCP traffic through the tunnel or use tun-mtu on the server with a size that is small enough to not have packets that are too large. e.g. try using tun-mtu 1380 then packets get split inside the tunnel.
Are you using mtu-disc yes in your config?
What you are hitting is "OpenVPN generated packets should be fragmented by the router" (given that the sending host is not aware that there is a 1492 stepdown in the path). If mtu-disc is active, OpenVPN will set the DF bit, and the router may not fragment, the sending OS should take not on this and do the fragmentation, but I'm not really sure what is happening without
spending some time looking at dumps and testing.
Pings larger than 1448 will create UDP packets > 1500, so the sending host knows "I can not send this" and fragments right away - depending on OS, you might see 2 half-sized packets, or 1 large + 1 small. 2 half-sized will get through the 1492 bottleneck just fine...
Long story short: MTU and VPNs is highly complex, and not all cases will "just work".
As @schwabe wrote, for TCP this is solvable by mssfix, for ICMP it can be worked around using fragment, or by lowering the inside (tun) MTU to 1400 or so - so, no 1493..1500 byte external packets can be created.
Thanks for you comments, I can see how this issue can actually create a situation where only a certain range of packet sizes will fail.
I don't have mtu-disc yes in any config, both client and server. I have actually never used this option since forever.
I get the idea that Openvpn believes that the path MTU is 1500 while it's actually 1492, but reading the error it logs, it seems that somehow Openvpn actually knows that the path MTU is 1492, it even prints it. Or is the OS that knows this?
In this case, using "fragment" will surely work, as probably will using link-mtu 1492 to tell openvpn that we have a limit down the line.. but in that case if the real path mtu is even lower, will it fail?
I have done some more tests.
If I use "mtu-disc yes" I get the same results but the error in the log now reports the MTU as 1500 and not 1492.
If I use "mtu-disc no" it works fine.
I don't get why... is it because now openvpn does never set the DF flag on its udp traffic at all, so everything gets fragmented as needed? By sniffing I can now see that UDP data from Openvpn gets indeed fragmented when needed.
Still I don't think I have understood the advantages and disadvantages of setting mtu-disc to yes or no and how should it actually work.
@Kurgan- MTU istuff is very complex. And the interaction what happens with DF and so on are something even not completely understand by us developers that are doing this stuff for 10+ years. I would suggest to add tun-mtu 1420 to your configs and be done with it.
Ok, I'll try it too. Thanks a lot. In fact what I'm really looking for is a config that I can count on in quite every situation, I really cannot debug each and every installation I happen to make. Thanks anyway, I'll keep all of this documentation for reference.