GrapheneOS
Apps can bypass VPN with killswitch by sending multicast packets
I have been trying out the security of the VPN functionality in GrapheneOS to see that there are no leaks or VPN bypass when killswitch is enabled. I have monitored the network traffic using Wireshark on my Wifi hotspot.
Here I have observed that apps are able to send multicast packets not going out over the VPN, but instead going to the local network, despite killswitch being enabled. This even with apps like Mullvad VPN app that clearly says it blocks all local network traffic. This is a clear privacy issue in the common case for using a VPN where the local network is untrusted, because the user expects no one on the network learning about your apps or usage when on the VPN. This issue is reproducible with both Mullvad VPN app and official Wireguard app, and probably others, so likely an issue in GrapheneOS rather than the apps.
Steps to reproduce:
- Install Mullvad VPN app from F-Droid or official Wireguard app from their website.
- Login to or configure your VPN app, and make sure always-on and killswitch is enabled for the VPN in settings.
- Install VLC app from F-Droid.
- Start VLC and click to menus to browse videos, audio and other media. Broadcast packets are being sent out on the local network to scan for media sources, bypassing the VPN.
The broadcast packets are regular IP broadcast packets of UDP and IGMP kind. I don't know if the app can receive answers to these broadcasts.
Possible relevant forum thread, Spotify may be sending multicast packets as well: https://discuss.grapheneos.org/d/10337-spotify-communicating-with-other-devices-on-local-network/6
Quote from John-longson:
Long story short, I made an anonymous Spotify account (that has never seen the internet without a vpn and has never interacted with another account) and was listening to music on WiFi when my girlfriend got home and turned on her Spotify and mine had a pop up asking me to join her session. This shouldn't be possible since we both use a VPN. I contacted the VPN and also tried other VPNs and the result is always the same. The VPN has LAN disabled, and all other settings set to avoid this, including killswitch.
Spotify must have found some vulnerability either in the OS or the networking and is able to see other devices on the network or has just enough data to fingerprint your network and discover if two Spotify devices are on the same network. I confirmed this by turning off WiFi, and her device disappeared.
This clearly illustrate the privacy violation happening, and what nefarious things apps can do that the user does not expect when distrusting the local network by using a VPN with killswitch. It was found later in the thread it likely is mDNS multicast packets doing this.
Quote from alfred:
I was not able to replicate getting the Spotify session to come up, but it could be my set up. Verified with a packet capture that Spotify does use mdns (Multicast DNS), which does not go over the VPN even if the kill switch and blocking LAN is enabled.
I have been digging a little bit into this issue. The source code suggests routing tables are modified using netlink to enable and disable the kill switch. It does not look like iptables or similar is being used to implement the kill switch at all, judging from the source code alone.
I could actually print all routing tables and rules without root permissions on the release build of the device, so I could look into this without having managed to set up my build environment yet. I couldn't see if iptables is used though, since that required root, but as seen below, the kill switch is indeed set up using routing rules.
Here is the output of running "ip rule" which list the rules for IPv4. The comments are mine.
# Loopback and broadcast permitted on all interfaces including real ones? Uncertain about what this is.
0: from all lookup local
# Dead rule, will be skipped
10000: from all fwmark 0xc0000/0xd0000 lookup 99
# This allows the system (uid=0) to send over Wifi (wlan0), it also sets Wifi as default route
11000: from all iif lo oif dummy0 uidrange 0-0 lookup 1002
11000: from all iif lo oif wlan0 uidrange 0-0 lookup 1046
# Dead rule, will be skipped
12000: from all iif tun0 lookup 97
# This allows all apps to send over VPN (tun0), it also sets VPN as default route
13000: from all fwmark 0x0/0x20000 iif lo uidrange 1300000-1399999 lookup 1051
13000: from all fwmark 0xc0067/0xcffff lookup 1051
# This is the kill switch that blocks the packet from being sent, unless sent by a rule above
14000: from all fwmark 0x0/0x20000 iif lo uidrange 1300000-1310152 prohibit
14000: from all fwmark 0x0/0x20000 iif lo uidrange 1310154-1320152 prohibit
14000: from all fwmark 0x0/0x20000 iif lo uidrange 1320154-1399999 prohibit
... there are many more rules, but regular app traffic should have been blocked at this point ...
Here are the actual routing tables:
$ ip route list table 1051
default dev tun0 proto static scope link
10.0.0.1 dev tun0 proto static scope link
$ ip route list table local
local 10.0.0.1 dev tun0 proto kernel scope host src 10.0.0.1
broadcast 127.0.0.0 dev lo proto kernel scope link src 127.0.0.1
local 127.0.0.0/8 dev lo proto kernel scope host src 127.0.0.1
local 127.0.0.1 dev lo proto kernel scope host src 127.0.0.1
broadcast 127.255.255.255 dev lo proto kernel scope link src 127.0.0.1
broadcast 192.168.0.0 dev wlan0 proto kernel scope link src 192.168.0.100
local 192.168.0.100 dev wlan0 proto kernel scope host src 192.168.0.100
broadcast 192.168.0.255 dev wlan0 proto kernel scope link src 192.168.0.100
I am wondering about those broadcast rules, do they permit multicast too maybe? I will try to dig a little bit deeper later, but this is as far as I got now.
@ryrona2 filtering for killswitch vpn is done via netd per UID not iptables. see https://github.com/GrapheneOS/platform_frameworks_base/blob/14/services/core/java/com/android/server/connectivity/Vpn.java#L2046
I can confirm that Spotify cannot detect a Spotify Connect enabled speaker on my home LAN unless I disable Always-On mode for the VPN, even though Local Network Access is always enabled from within the Mullvad app.
Perhaps the multicast discovery packets may be making it out (hence the popup that the forum poster saw from his girlfriend's Spotify) but the replies are being lost?
FYI, Spotify Lite is much less bloated (and likely more privacy-friendly), but with limited functionality (such as no Spotify Connect).
@maade93791 Yes. I have gone down that path already, and then to netd, and there I saw that netd sets up the kill switch using netlink to manipulate the routing table rules, and nothing else. So, I do not believe iptables is being used either.
@thestinger Really? I felt pretty confident I have found the actual logic being used, and that was manipulation of routing table rules. But I guess I will know soon anyway, I have made a userdebug build and am trying to figure out how one actually flashes the thing. After that I should be easily able to confirm whether I have found the right place or not by simply manually removing the routing table rules I believe is the killswitch and see what happens.
@no-usernames-left That could be so. I know the packets are definitely sent out, but not more than that. And since it isn't iptables being used, it is a bit hard for me to understand what the rules actually do block and not. But I will see if I can figure it out. It is a bit odd they didn't go with iptables, since iptables is designed to support use-cases like this.
@ryrona2 I wonder if the issue is not related to the VPN kill switch. It sounds like eBPF is used to filter/tag traffic, so I wonder if multicast traffic is not evaluated in the eBPF program, or maybe not assigned to the user profile.
multicast range is 224.0.0.0/4, so one potential solution would be to block that range as part of the VPN kill switch.
I think the issue is likely with the eBPF code. Linux tries VERY hard to receive and send packets. It will receive and send them in essentially any way by default. Filtering has to be implemented via eBPF or netfilter (iptables/nftables). It appears Android is using eBPF for this right now. If you looked at the iptables rules and determined it wasn't done there, that doesn't mean it's done with route configuration.
On our servers, we use nftables to emulate a strong host model for input, but not currently output, and we exclude loopback since there are many things requiring the weak host model for that including some of the services we run and the synproxy functionality we use as part of DDoS protection:
https://github.com/GrapheneOS/infrastructure/blob/3b1c43d29fa4132cf44ac33126d25bbfe31187c9/nftables/nftables-ns1.conf#L35-L38
@alfred6427 @thestinger There is something in the routing rules about tagged packets, so if either of you know where the eBPF code is, I would appreciate if you could give me a hint. I cannot promise I will get any time to look into this soon though, so anyone else is free to pick up the task. I don't know a think about eBPF, even if I know iptables/nftables well, so it would probably be faster for someone else to find the issue anyway. Or maybe we should just report it upstream.
@ryrona2 I don't know much about eBPF or where the code would be. I think they are typically .bp extension.
We have a partial fix for this implemented. It blocks apps sending or receiving multicast packets. However, it doesn't yet block the kernel generated IGMP packets triggered by apps. There may also be kernel generated MLD packets for IPv6.
It's not merged yet since some tweaks may be required. This issue will be closed when it's landed and we'll open a new issue for the remaining IGMP and potentially also MLD issue. We can probably have a single issue for both ICMP and MLD if applicable since the fix likely wouldn't be specific to one or the other.
This is now fully fixed by the combination of https://github.com/GrapheneOS/platform_packages_modules_Connectivity/commit/615c33e677bd19ee023178e4aab11c43989123c7 and https://github.com/GrapheneOS/platform_system_netd/commit/61811e6b628b5183375a516ab4328edb2393b29b.
Fixing this was much more involved than we had expected. We needed eBPF enforcement to address what was reported here but we discovered other issues requiring more complexity for the eBPF enforcement. We also discovered an issue where users could send multicast via each other's VPN which we had to address via iptables since it wasn't clear how to do that via eBPF. It's a lot more code than anticipated but it's not at all invasive and should hopefully be easy to keep maintained and ported to new versions. We're not going to try to upstream it in the near future.
These are the relevant release notes:
- extend standard Android eBPF filter to prevent apps sending multicast packets outside of the VPN tunnel either directly or separately via kernel-generated multicast traffic (IGMP, MLD) when leak blocking is enabled
- add netfilter-based multicast firewall only permitting sending multicast packets to permitted interfaces for the process
This shouldn't be able to cause any compatibility issues as we experienced with DNS leak blocking. We need to revisit the DNS leak blocking to make it stricter while avoiding the app compatibility issues caused by our initial approach next.
This caused minor app compatibility issues we can likely easily resolve and unfortunately major carrier/network compatibility issues which weren't reported during ~20 hours of Beta testing so we need to rush out a new release reverting these changes and we have a big support issue to deal with. This is unfortunately likely going to be delayed until after Android 15 and we're going to have to be very cautious about shipping it.