UDP ports not forwarded to host

[ollieayre]

Hi

I'm running lima v0.7.2 and can't seem to get UDP ports to forward to the host as TCP ports are.

Is this a known issue/expected behaviour?

I'm trying to run PiHole using the following command which successfully forwards both TCP and UDP ports when run using docker-for-mac so i'm fairly confident i've got that right.

me@home ~ % lima nerdctl run -d \
    --name pihole \
    -p 53:53/tcp \
    -p 53:53/udp \
    -p 67:67/udp \
    -p 8080:80 \
    -e TZ="GMT" \
    -v "<base>/opt/pihole/etc-pihole/:/etc/pihole/" \
    -v "<base>/opt/pihole/etc-dnsmasq.d/:/etc/dnsmasq.d/"
    --dns=127.0.0.1 \
    --dns=1.1.1.1 \
    --restart=always \
    --hostname pi.hole \
    --cap-add=NET_ADMIN \
    -e VIRTUAL_HOST="pi.hole" \
    -e PROXY_LOCATION="pi.hole" \
    -e ServerIP="0.0.0.0" \
    pihole/pihole:latest

Once up and running this is what i get when checking ports on the guest and then on the host:

me@lima-default: $ lsof -i:53
COMMAND   PID       USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
rootlessk 897 me   12u  IPv6 149573      0t0  TCP *:domain (LISTEN)
rootlessk 897 me   14u  IPv6 148128      0t0  UDP *:domain

me@home ~ % lsof -i:53
COMMAND  PID       USER   FD   TYPE             DEVICE SIZE/OFF NODE NAME
ssh     1969 me   18u  IPv4 0xc3df8c0afd3b70a9      0t0  TCP *:domain (LISTEN)

Any help much appreciated as this really isn't my area of expertise.

Thanks

[jandubois]

Is this a known issue/expected behaviour?

Yes, port forwarding is implemented via ssh tunneling, which only supports TCP.

So we would need to have a way to forward UDP traffic to TCP, send it over the tunnel and convert it back to UDP. This may be complex, as we would also have to make sure UDP datagrams aren't being split into multiple packets (or if they are, we would have to re-assemble them on the other side).

Maybe somebody else will come up with an easier way to do this...

[afbjorklund]

I think that is how you do it, possibly you can use socat to set it up ?

[buu700]

I'm also seeing that this should be possible with netcat or socat:

• https://www.disk91.com/2020/technology/systems/transfer-udp-over-ssh

• https://www.quora.com/How-do-you-forward-UDP-packets-through-an-SSH-tunnel

---

I'm not super familiar with either tool and have never done this before, but based on the first link it sounds like this is all that's needed:

• Inside the VM: socat tcp4-listen:${SSH_TUNNEL_PORT},reuseaddr,fork UDP:${HOST_IP}:${UDP_PORT}

• On the host: socat -T15 udp4-recvfrom:${UDP_PORT},reuseaddr,fork tcp:localhost:${SSH_TUNNEL_PORT}

[jammy-d]

I'm not greatly familiar with either socat or netcat, but after spending a good part of a day looking at both workarounds, unfortunately I think there are problems getting either to work reliably for my use case.

socat seemed promising at first - several tests I did on an Oracle Linux container worked absolutely as expected. But I'm now convinced the OS X version of socat installed using brew is bugged.

Specifically it boils down to socat running with the udp-recvfrom option - it just ends up killing itself when there are more than several messages sent in quick succession. It also doesn't work with multiple UDP clients sending messages to the same UDP port. Both these problems didn't occur in the Oracle Linux version I tested.

The problem is reproducible through a simple test when running in Terminal:

• The client: socat - udp-sendto:127.0.0.1:${UDP_PORT}
• The server: socat udp-recvfrom:${UDP_PORT},reuseaddr,fork -

In the client side, if you mash the enter key the server eventually falls over. You can see the process ID disappearing with a ps -ef | grep socat and the port disappears looking at netstat -anv | grep ${UDP_PORT}.

An odd observation is that on OS X, every time I hit enter in the client side there seemed to be a chance of a random number of socat processes spawning in ps, and according to netstat the process that was listening on the port seemed to change around (although sometimes I couldn't even find the process running with ps). On Oracle Linux running the exact same test, it doesn't appear to spawn additional processes, or change the process that is listening on the port around.

When trying the 2 clients, the messages from the second client that sent the UDP message completely get ignored on the server, and not only that but it also stops getting some of the messages from the first client. This also worked fine on Oracle Linux.

I tried with the udp-listen and udp options, the issue with the mashing of the enter key isn't reproducible, but it doesn't allow more than one client to connect without restarting the server.

Trying a similar test with netcat, both the nc on OS X (default one on OS X) and the nc ran from Oracle Linux (nmap-ncat) don't allow more than one client to communicate with the server without restarting the server. It seems as though the first server stops listening on ports other than on the specific source port of the first client that sent it a UDP message. This was tried with and without the -k flag on the server (which wasn't accepted on nmap-ncat with UDP, and didn't seem to change the behaviour on the default OS X nc).

I wanted to test the netcat version from OpenBSD on Oracle Linux but unfortunately it's not easy to install with the aarch64 architecture container (running on an M1 Mac).

[jammy-d]

Just correcting myself slight, the issues with socat is seen on a newer model of the MacBook Pro (M1 model) which is running socat 1.7.4.3. I've just tried the same experiment with an older MacBook Pro with an Intel CPU running socat 1.7.3.4, and neither issue occurs, so this could be related to the latest socat version.

[buu700]

Interesting, thanks for investigating all that @jammy-d! It sounds like you've basically validated the concept, M1 socat bugs aside.

Rather than socat, maybe an alternative like sslh would work, although now that I think about it in either case Lima would need to work around the GPL licensing of those projects in some way.

The fastest and most surefire way is probably just to write some custom code that does exactly what's needed. I know offhand that it's simple to write UDP and TCP clients and servers with Node.js, and I imagine that the Go standard library would allow for the same if that were preferable.

An implementation could look something like:

• Inside the VM: node -e "const client = dgram.createSocket('udp4') ; client.connect(${UDP_PORT}, () => net.createServer(server => server.on('data', data => client.send(data, 0, data.length))).listen(${SSH_TUNNEL_PORT}, '${HOST_IP}'))"

• On the host: node -e "const client = net.createConnection(${SSH_TUNNEL_PORT}, () => dgram.createSocket('udp4', message => client.write(message)).bind(${UDP_PORT}))"

(Haven't tested either of those lines, but they should work as-is.)

[buu700]

@jandubois Is there any obvious reason why this approach shouldn't work? I know you'd brought up reassembling UDP datagrams, but TCP and UDP have the same max packet size (64 KiB), so I think forwarding the packets agnostically should actually just work as expected.

Would it be a major change to put Go equivalents to those two lines in guestagent and hostagent respectively? I would do it myself and send a PR (if it didn't take too much time), but I'm not entirely sure where to start looking for the code that handles the port arguments from nerdctl.

[jandubois]

I know you'd brought up reassembling UDP datagrams, but TCP and UDP have the same max packet size (64 KiB), so I think forwarding the packets agnostically should actually just work as expected.

Idk, I can't concentrate on this right now. You would probably have to set the MSS value in the TCP connection options to avoid fragmentation; I don't know what it defaults to when there is no link-level MTU.

@AkihiroSuda what do you think?

Would it be a major change to put Go equivalents to those two lines in guestagent and hostagent respectively?

I think that would be fine, but it looks like your example only forwards UDP from the VM to the host, but not the other way.

[buu700]

I think that would be fine, but it looks like your example only forwards UDP from the VM to the host, but not the other way.

The other direction actually already seems to work, in my testing.

[jammy-d]

Just following up on what I wrote regarding the socat workaround, I'm pretty sure the issue is with socat version 1.7.4.x. Downgraded socat on the Mac to 1.7.3.4 and it seems to work fine for me.

[danieldonoghue]

for additional info/workarounds..

https://superuser.com/questions/53103/udp-traffic-through-ssh-tunnel

[cwash]

Are there plans to address this issue directly - versus the various workarounds that are out there? Does addressing this issue require reimplementing the entire port-forwarding feature?

[zmaktouf]

I am hitting this limitation. We really need this ticket to be prioritized. Thanks

[agjmills]

👍 for udp forwarding to the host

[AkihiroSuda]

Workaround: use vmnet to assign a "real" IP https://github.com/lima-vm/lima/blob/v0.14.1/examples/vmnet.yaml

[blond-in-blue]

Workaround: use vmnet to assign a "real" IP https://github.com/lima-vm/lima/blob/v0.14.1/examples/vmnet.yaml

Thank you! Setting the network to "bridged" and forwarding the ports from the separate lima-* hostname on my network worked for my case. Would still enjoy having UDP forwarding to the host itself but for now this is a good workaround.

[Digicrat]

Any updates?

At the very least, would it be possible to get Docker under [co]lima to report an error if a user tries forwarding a UDP port when it's not supported? I just wasted a significant amount of time trying to debug my system before finding this ticket explaining that it's not supported and significantly complicated what was supposed to be an easy offline test environment.

[leonboot]

I'm glad the thought of possible issues with UDP popped into my head early on when I hit this snag. Googling it quickly led me to this issue :-) I could have seen myself spending a lot of time just like @Digicrat. FWIW, a +1 for me on out-of-the-box UDP forwarding!