tonarino
Feature: Assigning external IP addresses (or address ranges) to a peer
Motivating example: I have an external network, and I want to make the external network and the innernet network fully routable between each other.
The idea I have is to use one innernet peer as a relay/router that forwards traffic between the two networks. I can set up the routes on my external network just fine, but the innernet peers also need to be configured to send traffic for the other network to the relay peer.
(For simplicity, let's assume the external network is on a separate subnet, not a part of the innernet subnet, and the only machine that is a member of both networks is the relay.)
To implement this, the additional addresses for the peer could be registered in the innernet server database, and then distributed to the other peers as "additional routes" of the relay peer. The innernet clients can add those addresses to the "Allowed IPs" list of the relay peer and add the corresponding routes in the OS network interface.
This is related to #2, #3, and a little bit of #210. If that external network I described before is a "vanilla Wireguard" network, managed separate from innernet, then this can give vanilla Wireguard clients (including Android) access to the innernet network. By using a relay server, the vanilla configuration stays simple and static, and doesn't require manual work or a daemon to maintain the full list of peers and do the whole NAT traversal dance. Most of the configuration is done once on the relay server.
Other alternatives: It is probably possible to use NAT on the relay, so all the traffic on the innernet side goes through the relay's primary innernet address. That has the usual drawbacks of NAT: only the external network can establish a connection to the innernet network, and not vice versa (besides port forwarding).
This is not an important feature for me right now, but the problem of android and vanilla connections has been in the back of my mind for a while, and I had this idea that I wanted to share.
If other people show some interest, then I wouldn't mind helping out. In the meantime, I may put together a repo with some scripts/configs for a relay server based on NAT.
I would like my phone on my innernet (so, really I'm interested in #3). Gluing in a router seemed like the most direct way. I prototyped this and got it working, but sketchily.
The only tricky part is that AllowedIPs of the router peer is always a single address instead of a subnet. If you assertively extend it on each innernet client it works.
tl;dr,
on the router:
sysctl net.ipv4.conf.all.forwarding=1
on every other peer:
ip route add $EXTERNAL_SUBNET via $ROUTER
(while true; do
wg syncconf $YOUR_INNERNET <(wg showconf $YOUR_INNERNET | sed '/AllowedIPs = $ROUTER\/32/ s/$/, $EXTERNAL_SUBNET/');
sleep 4;
done)
Prototype
innernet
I set up an innernet on 10.13.0.0/16
$ sudo innernet
network: chapiteau
listening port: 57879
peer: innernet-server (e+vchtZGvh...)
ip: 10.13.0.1
endpoint: REDACTED:1337
last handshake: 1 minute, 3 seconds ago
transfer: 5.87 KiB received, 4.36 KiB sent
peer: desktop (YtNRrP9CI2...)
ip: 10.13.17.13
peer: laptop (4q61rgnQu7...)
ip: 10.13.17.17
endpoint: REDACTED:57425
last handshake: 37 seconds ago
transfer: 180 B received, 896 B sent
phones
I set up a "vanilla" Wireguard subnet on 10.14.0.0/16.
I chose innernet-server to double as my router. It has
# sysctl net.ipv4.conf.all.forwarding=1
I used wg genkey and wg pubkey to set up:
# /etc/wireguard/phones.conf
[Interface]
Address = 10.14.0.1/16
ListenPort = 7331
PrivateKey = REDACTED_SERVER_PRIVATE_KEY
[Peer]
PublicKey = pAH3iLXdubohtrn7xk1XOaVUhh8H1eWQ1Mub5OjG9FA=
AllowedIps = 10.14.23.23/32
and turned it on with wg-quick up phones.
On the other side I scanned this config into the Wireguard app (via QR code):
[Interface]
Address = 10.14.23.23/32
PrivateKey = REDACTED_PHONE_PRIVATE_KEY
[Peer]
PublicKey = 6yJ7ypB9f+AcE0p2N6vtmbTCGxQpyh+jx9ZWRguIHF0=
Endpoint = REDACTED:7331
AllowedIPs = 10.13.0.0/16, 10.14.0.0/16
which turned itself on.
(The PublicKeys each correspond to the other's PrivateKey, I hope the redactions aren't too confusing)
routing
It's 90% of the way there. If I ping 10.13.17.17 from the phone and watch tcpdump -i phones and tcpdump -i chapiteau on the router I can see that the packets are making it from 10.14.23.23 to the router. But they're not showing up in tcpdump on laptop
To debug, I turned on
echo "module wireguard +p" | sudo tee /sys/kernel/debug/dynamic_debug/control
and found
Nov 19 15:04:50 laptop kernel: wireguard: chapiteau: Packet has unallowed src IP (10.14.23.23) from peer 4 (REDACTED:1337)
because laptop thinks that (and every) peer only gets a single IP:
laptop# wg
peer: e+vchtZGvhvt7LtIWLgjBfzbjNzdAXZR+UvmKhJN4zc=
endpoint: REDACTED:1337
allowed ips: 10.13.0.1/32
To fix this, first add a cross-subnet route:
laptop# ip route add 10.14.0.0/16 via 10.13.0.1
And append the same subnet to Wireguard so it permits packets to pass
laptop# (while true; do
wg syncconf chapiteau <(wg showconf chapiteau | sed '/AllowedIPs = 10.13.0.1\/32/ s/$/, 10.14.0.0\/16/');
sleep 4;
done)
Now I can ping both ways!
But this is the sketchy part: it has to fight innernet and it has to be running on all peers. And would be better if run as a system service not just a script I've pasted into my shell.
Demo
With this, I can, e.g. run sv start sshd in Termux and log in over the VPN:
laptop$ ssh -p 8022 10.14.23.23
[email protected]'s password:
Welcome to Termux!
Docs: https://termux.dev/docs
Donate: https://termux.dev/donate
Community: https://termux.dev/community
If I watch tcpdump -i chapiteau on the router it's clear that there's traffic only when I type into this ssh session, so the router is indeed routing.
Design
I'm not sure how to make this elegant. It gets tangly as soon as I start thinking about it.
Should innernet detect when it's told to act as a router and enable (or disable?!) net.ipv4.conf.all.forwarding?
I'd suggest just calling the "additional routes" AllowedIPs in the peers table, to avoid adding new terminology. On the other hand, there's already a subnet concept in innernet, the cidrs table, maybe a peer should have a list of cidrs that it routes for? But then you wouldn't be able to interconnect non-innernets, unless you modelled the non-innernet as a CIDR in your innernet. Ah. It's tricky.
Solution via NAT
I simplified my solution to make innernet-server NAT from phones -> chapiteau.
This isn't fully routable, but it lets my phone reach my file server which is really all I want. I bet it's all you want too. The only reason I ever connect laptop -> phone is to get photos off via adb pull, but with this enabled it's simpler to transfer them phone -> laptop (either Termux with scp, or using the sftp support in Material Files).
This set up doesn't need any extra routing on entry on every peer nor weird loops fighting against innernet to keep it stable. It just uses standard kernel features, configured once.
Again, set up innernet on 10.13.0.0/16, hosted by server:
server# cat /etc/innernet-server/chapiteau.conf
private-key = "REDACTED"
listen-port = 1337
address = "10.13.0.1"
network-cidr-prefix = 16
desktop# innernet
network: chapiteau
listening port: 55480
peer: innernet-server (e+vchtZGvh...)
ip: 10.13.0.1
endpoint: REDACTED:1337
last handshake: 2 seconds ago
transfer: 1405.38 KiB received, 782.55 KiB sent
peer: desktop (YtNRrP9CI2...)
ip: 10.13.17.13
endpoint: REDACTED
last handshake: 1 minute, 49 seconds ago
transfer: 49.42 MiB received, 613.78 KiB sent
peer: laptop (4q61rgnQu7...)
ip: 10.13.17.17
Set up wg-quick on 10.14.0.0/16 on innernet-server:
# cat /etc/wireguard/phones.conf
[Interface]
Address = 10.14.0.1/16
ListenPort = 7331
PrivateKey = REDACTED
[Peer]
PublicKey = 9nTByJDjfeZv0T4dNcTInxuYvWJUxKRomdrrPXYhuTM=
AllowedIps = 10.14.17.23/32
Set up vanilla Wireguard app on Android, using this config:
[Interface]
Address = 10.14.17.23/32
PrivateKey = REDACTED_PHONE_PRIVATE_KEY
[Peer]
PublicKey = e+vchtZGvhvt7LtIWLgjBfzbjNzdAXZR+UvmKhJN4zc=
Endpoint = REDACTED:7331
AllowedIPs = 10.13.0.0/16, 10.14.0.0/16
PersistentKeepalive = 25
Route, but only between the VPNs [^1] with a NAT between them, treating innernet like you'd usually NAT the internet.
[^1]: turning on sysctl net.ipv4.conf.all.forwarding doesn't make you an open proxy because return traffic won't come back through you, but I think it makes it possible to attackers to bounce UDP off you to harass their targets.
server# cat /etc/sysctl.d/50-innernet-NAT.conf
net.ipv4.conf.chapiteau.forwarding=1
net.ipv4.conf.phones.forwarding=1
server# cat /etc/iptables/iptables.rules
# I made figured this out with iptables-save
*nat
# I don't know why I have to express VPN B as an IP address and VPN A as an interface name here, but that's how iptables works. I guess that's why everyone wants to get rid of iptables.
-A POSTROUTING -s 10.14.0.0/16 -o chapiteau -j MASQUERADE
COMMIT
Turn it all on:
sysctl --system
systemctl enable --now wg-quick@phones innernet-server@chapiteau iptables
When this is up, this is what server sees:
# wg
interface: chapiteau
public key: e+vchtZGvhvt7LtIWLgjBfzbjNzdAXZR+UvmKhJN4zc=
private key: (hidden)
listening port: 1337
peer: YtNRrP9CI2I4T3AgKt54r5myzRpZPykegaQINdId71o=
endpoint: REDACTED
allowed ips: 10.13.17.13/32
latest handshake: 24 seconds ago
transfer: 1.31 GiB received, 62.74 MiB sent
persistent keepalive: every 25 seconds
peer: 4q61rgnQu7woVKVolnL9L9xiXURxxACTlM5eqsd8rUs=
endpoint: REDACTED
allowed ips: 10.13.17.17/32
latest handshake: 1 minute, 46 seconds ago
transfer: 1.22 MiB received, 4.72 MiB sent
persistent keepalive: every 25 seconds
interface: phones
public key: e+vchtZGvhvt7LtIWLgjBfzbjNzdAXZR+UvmKhJN4zc=
private key: (hidden)
listening port: 7331
peer: 9nTByJDjfeZv0T4dNcTInxuYvWJUxKRomdrrPXYhuTM=
endpoint: REDACTED
allowed ips: 10.14.17.23/32
latest handshake: 42 seconds ago
transfer: 57.09 MiB received, 1.31 GiB sent
Note that the two wg interfaces on server have the same public key. It's apparently fine to do this. You don't have to, you could wg genkey twice. I think it adds complexity for not much security, but maybe I'm wrong.
With this my phone can ping 10.13.17.13, ping 10.13.17.17, use sftp in Material Files to access files on both of those machines -- videos even stream mostly fine from sftp -> Material Files -> VLC --, or access my management console at http://10.13.17.13:9091. Everything is getting bounced through server, a cheap VPS, but honestly I barely notice; the latency is 50ms and once a transfer gets going the bandwidth maxes out at my home connection's bandwidth.
Mesh
I am still interested in being able to glue networks together over innernet. What I wrote above is just a very reliable a workaround. This isn't at all to devalue @agausmann's request for full routability.