setup-ipsec-vpn Return traffic from local DNS container not traversing VPN tunnel

Return traffic from local DNS container not traversing VPN tunnel

Open diamondsw opened this issue 2 years ago • 1 comments

Summary: VPN and DNS containers on the same host; both work as expected independently, but DNS return traffic does not reach VPN clients. Seems like a docker networking issue, but I'm at my wits' end.

This is going to be a bit of a weird one; thank you for looking! Despite the amount of DNS detail, I swear it's not a DNS issue. The goal here is to provide an internal DNS server for private hostnames on the remote network. While this is my first time using this container, I've configured (painfully) several IPSEC/L2TP VPNs before, with mobileconfig, site-to-site tunnels, and private DNS. First time I've ever seen something quite like this, and I'm completely stumped.

Checklist

[x] I read the README
[x] I read the Important notes
[x] I followed instructions to configure VPN clients
[x] I checked Troubleshooting, IKEv2 troubleshooting and VPN status
[x] I searched existing Issues
[ ] This bug is about the VPN setup scripts, and not IPsec VPN itself

I don't know on that last one! If anything, it seems like a very weird Docker network thing.

Describe the issue After establishing a VPN connection to the ipsec-vpn-server container, return traffic from a dnsmasq container on the same host does not arrive.

More detail below, but in short:

The VPN connects without issue (by the way, BRAVO at making IPSEC so easy!)
Ping, SSH, and other bidirectional traffic to the host server or internet traverse the VPN without issue.
If no DNS server is specified for the VPN (i.e. using built-in 8.8.8.8), then DNS resolution works
If a public DNS server is specified (e.g. 1.1.1.1), then DNS resolution works
If a local dnsmasq container is specified using the host private IP address, then dnsmasq sees the requests and responds - but the client never sees these responses
Meanwhile, accessing the same DNS container over the internet works, and using dig locally on the server works

That's the weirdest thing - the DNS container works, and even sees the requests over the VPN and responds - but the client never gets the traffic back. Other protocols are working fine. The only difference I can see is this is traffic between containers, whereas ping, ssh, etc are all either to the host or headed outside the network.

Network details:

Server is on Oracle Cloud and has both a public IP (141.147.x.y - redacted) and private IP (10.2.1.8), and the security list allows USP 500/4500 from all IPs. For the purposes of troubleshooting, it also allows port 53 from my home IP.
The public IP is reachable at 'relay.redacted.com', and the private IP at 'relay.oracle.redacted.com'. Apologies if this is confusing!

To Reproduce Steps to reproduce the behavior:

Setup the VPN server with the following docker-compose.yml:

version: "3"
services:
  ipsec-vpn:
    image: hwdsl2/ipsec-vpn-server:latest
    container_name: ipsec-vpn
    restart: unless-stopped
    environment:
      VPN_IKEV2_ONLY: 'yes'
      VPN_DNS_NAME: 'relay.redacted.com'
      VPN_DNS_SRV1: '10.2.1.8'
    ports:
      - '500:500/udp'
      - '4500:4500/udp'
    volumes:
      - /lib/modules:/lib/modules:ro
      - ./data:/etc/ipsec.d
    cap_add:
      - NET_ADMIN
    devices:
      - "/dev/ppp:/dev/ppp"
    sysctls:
      - net.ipv4.ip_forward=1
      - net.ipv4.conf.all.accept_redirects=0
      - net.ipv4.conf.all.send_redirects=0
      - net.ipv4.conf.all.rp_filter=0
      - net.ipv4.conf.default.accept_redirects=0
      - net.ipv4.conf.default.send_redirects=0
      - net.ipv4.conf.default.rp_filter=0
      - net.ipv4.conf.eth0.send_redirects=0
      - net.ipv4.conf.eth0.rp_filter=0

Setup the DNS server with the following docker-compose.yml and config files:

Dockerfile

FROM alpine:edge
RUN apk --no-cache add dnsmasq
EXPOSE 53 53/udp
ENTRYPOINT ["dnsmasq", "-k", "--log-facility=-"]

docker-compose.yml

version: "3"
services:
  dnsmasq:
    build: .
    image: diamondsw/dnsmasq
    container_name: dnsmasq
    restart: unless-stopped
    environment:
      TZ: 'America/Chicago'
    ports:
      - "53:53/udp"
    volumes:
      - ./dnsmasq.conf:/etc/dnsmasq.d/dnsmasq.conf
      - ./hosts:/etc/dnsmasq.d/hosts
    cap_add:
      - NET_ADMIN

dnsmasq.conf

# Listen on the Docker eth0
interface=eth0

# Standard options
no-resolv
cache-size=10000
local-ttl=2

# Internal DNS
domain-needed
bogus-priv
domain=oracle.redacted.com
expand-hosts
addn-hosts=/etc/dnsmasq.d/hosts

# Forward all public queries to the internet
server=1.1.1.1
server=8.8.8.8

# Debugging
log-queries

Note here that I have specified an interface to listen to, which turns off the dnsmasq '--local-service' option.

hosts

10.2.1.8        relay.oracle.redacted.com

(Note, I have changed the domain names to remove private information.)

Expected behavior DNS responses should arrive back to the requestor (i.e. the VPN client machine).

Logs Check logs and VPN status, and add error logs to help explain the problem, if applicable.

DNSmasq log when VPN connected (with a LOT of Apple services connection attempts cut out):

me@kelvin ~ % ssh 10.2.1.8
me@relay ~ $ docker logs --follow dnsmasq
dnsmasq[1]: started, version 2.86 cachesize 10000
dnsmasq[1]: compile time options: IPv6 GNU-getopt no-DBus no-UBus no-i18n no-IDN DHCP DHCPv6 no-Lua TFTP no-conntrack ipset auth no-cryptohash no-DNSSEC loop-detect inotify dumpfile
dnsmasq[1]: using nameserver 1.1.1.1#53
dnsmasq[1]: using nameserver 8.8.8.8#53
dnsmasq[1]: read /etc/hosts - 7 addresses
dnsmasq[1]: read /etc/dnsmasq.d/hosts - 1 addresses
dnsmasq[1]: config 192.168.43.10 is NXDOMAIN
dnsmasq[1]: config 192.168.43.10 is NXDOMAIN
dnsmasq[1]: config 192.168.43.10 is NXDOMAIN
dnsmasq[1]: query[A] relay2.oracle.redacted.com from 172.22.0.1
dnsmasq[1]: forwarded relay2.oracle.redacted.com to 8.8.8.8
dnsmasq[1]: reply relay2.oracle.redacted.com is NODATA-IPv4
dnsmasq[1]: config 192.168.43.10 is NXDOMAIN
dnsmasq[1]: query[A] relay.redacted.com from 172.22.0.1
dnsmasq[1]: forwarded relay.redacted.com to 8.8.8.8
dnsmasq[1]: reply relay.redacted.com is 141.147.x.y
dnsmasq[1]: query[A] relay.redacted.com from 172.22.0.1
dnsmasq[1]: cached relay.redacted.com is 141.147.x.y
dnsmasq[1]: query[A] relay.redacted.com from 172.22.0.1
dnsmasq[1]: cached relay.redacted.com is 141.147.x.y
dnsmasq[1]: config 192.168.43.10 is NXDOMAIN
dnsmasq[1]: query[A] relay.redacted.com from 172.22.0.1
dnsmasq[1]: cached relay.redacted.com is 141.147.x.y
dnsmasq[1]: query[A] www.google.com from 172.22.0.1
dnsmasq[1]: forwarded www.google.com to 8.8.8.8
dnsmasq[1]: reply www.google.com is 142.250.206.228
dnsmasq[1]: query[A] www.google.com from 172.22.0.1
dnsmasq[1]: cached www.google.com is 142.250.206.228
dnsmasq[1]: query[A] www.google.com from 172.22.0.1
dnsmasq[1]: cached www.google.com is 142.250.206.228
dnsmasq[1]: config 192.168.43.10 is NXDOMAIN
dnsmasq[1]: config 192.168.43.10 is NXDOMAIN
dnsmasq[1]: config 192.168.43.10 is NXDOMAIN
dnsmasq[1]: config 192.168.43.10 is NXDOMAIN
dnsmasq[1]: config 192.168.43.10 is NXDOMAIN
dnsmasq[1]: config 192.168.43.10 is NXDOMAIN
dnsmasq[1]: config 192.168.43.10 is NXDOMAIN
dnsmasq[1]: config 192.168.43.10 is NXDOMAIN
dnsmasq[1]: query[A] relay.oracle.redacted.com from 172.22.0.1
dnsmasq[1]: /etc/dnsmasq.d/hosts relay.oracle.redacted.com is 10.2.1.8
dnsmasq[1]: config 192.168.43.10 is NXDOMAIN
dnsmasq[1]: query[A] relay.oracle.redacted.com from 172.22.0.1
dnsmasq[1]: /etc/dnsmasq.d/hosts relay.oracle.redacted.com is 10.2.1.8
dnsmasq[1]: query[A] relay.oracle.redacted.com from 172.22.0.1
dnsmasq[1]: /etc/dnsmasq.d/hosts relay.oracle.redacted.com is 10.2.1.8
dnsmasq[1]: query[A] relay.oracle.redacted.com from 172.22.0.1
dnsmasq[1]: /etc/dnsmasq.d/hosts relay.oracle.redacted.com is 10.2.1.8
dnsmasq[1]: query[A] relay.oracle.redacted.com from 172.22.0.1
dnsmasq[1]: /etc/dnsmasq.d/hosts relay.oracle.redacted.com is 10.2.1.8
dnsmasq[1]: query[A] relay.oracle.redacted.com from 172.22.0.1
dnsmasq[1]: /etc/dnsmasq.d/hosts relay.oracle.redacted.com is 10.2.1.8
dnsmasq[1]: query[A] relay.oracle.redacted.com from 172.22.0.1
dnsmasq[1]: /etc/dnsmasq.d/hosts relay.oracle.redacted.com is 10.2.1.8
dnsmasq[1]: config 192.168.43.10 is NXDOMAIN
dnsmasq[1]: query[A] relay.oracle.redacted.com from 172.22.0.1
dnsmasq[1]: /etc/dnsmasq.d/hosts relay.oracle.redacted.com is 10.2.1.8
^C
me@relay ~ $

My dig attempts show up in the dnsmasq log as these lines:

dnsmasq[1]: query[A] relay.oracle.redacted.com from 172.22.0.1
dnsmasq[1]: /etc/dnsmasq.d/hosts relay.oracle.redacted.com is 10.2.1.8

Nothing much to show on the VPN side - it's happy:

me@relay vpn $ docker logs ipsec-vpn

VPN credentials not set by user. Generating random PSK and password...

Setting DNS server to 10.2.1.8...

Note: Running in IKEv2-only mode via env file option.
      IPsec/L2TP and IPsec/XAuth ("Cisco IPsec") modes are disabled.

Starting IPsec service...

Setting up IKEv2. This may take a few moments...

================================================

IKEv2 setup successful. Details for IKEv2 mode:

VPN server address: relay.redacted.com
VPN client name: vpnclient

Client configuration is available inside the
Docker container at:
/etc/ipsec.d/vpnclient.p12 (for Windows & Linux)
/etc/ipsec.d/vpnclient.sswan (for Android)
/etc/ipsec.d/vpnclient.mobileconfig (for iOS & macOS)

Next steps: Configure IKEv2 clients. See:
https://vpnsetup.net/clients2

================================================

xl2tpd[1]: Not looking for kernel SAref support.
xl2tpd[1]: Using l2tp kernel support.
xl2tpd[1]: xl2tpd version xl2tpd-1.3.17 started on 51a8334895ce PID:1
xl2tpd[1]: Written by Mark Spencer, Copyright (C) 1998, Adtran, Inc.
xl2tpd[1]: Forked by Scott Balmos and David Stipp, © 2001
xl2tpd[1]: Inherited by Jeff McAdams, © 2002
xl2tpd[1]: Forked again by Xelerance (www.xelerance.com) (C) 2006-2016
xl2tpd[1]: Listening on IP address 0.0.0.0, port 1701

Server (please complete the following information)

OS: Debian 11.5, running the cloudinit qcow2 image, but with the standard linux-image-arm64 kernel for ppp support
Hosting provider (if applicable): Oracle Cloud, free tier

Client (please complete the following information)

Device: MacBook Air M1
OS: macOS 12.6 "Monterey"
VPN mode: IKEv2

Client (please complete the following information)

Device: iPhone X
OS: iOS 16.0
VPN mode: IKEv2

Additional context

Connecting to the DNS over the internet (restricted to allow my IP only; no open resolvers):

me@kelvin ~ % dig @relay.redacted.com relay.oracle.redacted.com

; <<>> DiG 9.10.6 <<>> @relay.redacted.com relay.oracle.redacted.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 31390
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;relay.oracle.redacted.com.	IN	A

;; ANSWER SECTION:
relay.oracle.redacted.com. 2	IN	A	10.2.1.8

;; Query time: 154 msec
;; SERVER: 141.147.x.y#53(141.147.x.y)
;; WHEN: Mon Sep 19 08:29:49 CDT 2022
;; MSG SIZE  rcvd: 72

Connecting to the DNS from the server itself (i.e. connecting to localhost):

me@kelvin ~ % ssh relay.redacted.com
me@relay ~ $ dig @127.0.0.1 relay.oracle.redacted.com

; <<>> DiG 9.16.27-Debian <<>> @127.0.0.1 relay.oracle.redacted.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 49168
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;relay.oracle.redacted.com.	IN	A

;; ANSWER SECTION:
relay.oracle.redacted.com. 2	IN	A	10.2.1.8

;; Query time: 0 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Sun Sep 18 14:53:19 CDT 2022
;; MSG SIZE  rcvd: 72

Connecting to the DNS once connected to the VPN (both macOS DNS resolution and dig): connect vpn

me@kelvin ~ % dig @10.2.1.8 relay.oracle.redacted.com 

; <<>> DiG 9.10.6 <<>> @10.2.1.8 relay.oracle.redacted.com
; (1 server found)
;; global options: +cmd
;; connection timed out; no servers could be reached
me@kelvin ~ % ping relay.oracle.redacted.com
ping: cannot resolve relay.oracle.redacted.com: Unknown host
me@kelvin ~ %

disconnect vpn

Nothing. Weird!

Sep 20 '22 23:09 diamondsw

setup-ipsec-vpn setup-ipsec-vpn copied to clipboard

Return traffic from local DNS container not traversing VPN tunnel

setup-ipsec-vpn
setup-ipsec-vpn copied to clipboard