docker-openvpn
docker-openvpn copied to clipboard
kylemanna
Having issues when running on kubernetes
I have a working configuration for this image initially generated with this configuration command:
ovpn_genconfig -u tcp://old.host -z -N -D -s 10.82.171.0/24 -p 'route 192.168.5.0 255.255.255.0
Everything was working on plain docker (actually with rancher1 as orchestrator) and when I connect to vpn from outside I'm able to ping and access any service in 192.168.5.0/24 which is the main purpose of having this VPN.
Now I'm moving everything to kubernetes (plus rancher2) and I migrated old volume to a kubernetes deployment with the same specs as before. I've just changed VPN subnet to be able to have both images running in parallel:
ovpn_genconfig -u tcp://new.host -z -N -D -s 172.16.33.0/24 -p 'route 192.168.5.0 255.255.255.0'
I'd expect everything works as before, however I'm not able to interact with 192.168.5.0/24 network anymore.
I spent a lot of time in debugging this but I wasn't able to make it working. I compared line by line client output and it's identical (except for the networks used), so the issue must be in server configuration.
Here's server log in the old image (working)
15/11/2019 17:24:34RTNETLINK answers: Operation not permitted
15/11/2019 17:24:34RTNETLINK answers: Operation not permitted
15/11/2019 17:24:34RTNETLINK answers: Operation not permitted
15/11/2019 17:24:36Enabling IPv6 Forwarding
15/11/2019 17:24:36sysctl: error setting key 'net.ipv6.conf.default.forwarding': Read-only file system
15/11/2019 17:24:36Failed to enable IPv6 Forwarding default
15/11/2019 17:24:36Failed to enable IPv6 Forwarding
15/11/2019 17:24:36sysctl: error setting key 'net.ipv6.conf.all.forwarding': Read-only file system
15/11/2019 17:24:36Fri Nov 15 16:24:36 2019 OpenVPN 2.3.7 x86_64-alpine-linux-musl [SSL (OpenSSL)] [LZO] [EPOLL] [MH] [IPv6] built on Jul 10 2015
15/11/2019 17:24:36Fri Nov 15 16:24:36 2019 library versions: OpenSSL 1.0.2e 3 Dec 2015, LZO 2.09
15/11/2019 17:24:36Fri Nov 15 16:24:36 2019 Diffie-Hellman initialized with 2048 bit key
15/11/2019 17:24:36Fri Nov 15 16:24:36 2019 Control Channel Authentication: using '/etc/openvpn/pki/ta.key' as a OpenVPN static key file
15/11/2019 17:24:36Fri Nov 15 16:24:36 2019 Outgoing Control Channel Authentication: Using 160 bit message hash 'SHA1' for HMAC authentication
15/11/2019 17:24:36Fri Nov 15 16:24:36 2019 Incoming Control Channel Authentication: Using 160 bit message hash 'SHA1' for HMAC authentication
15/11/2019 17:24:36Fri Nov 15 16:24:36 2019 Socket Buffers: R=[87380->131072] S=[16384->131072]
15/11/2019 17:24:36Fri Nov 15 16:24:36 2019 ROUTE_GATEWAY 192.168.5.1/255.255.255.0 IFACE=eth0 HWADDR=66:32:30:39:36:37
15/11/2019 17:24:36Fri Nov 15 16:24:36 2019 TUN/TAP device tun0 opened
15/11/2019 17:24:36Fri Nov 15 16:24:36 2019 TUN/TAP TX queue length set to 100
15/11/2019 17:24:36Fri Nov 15 16:24:36 2019 do_ifconfig, tt->ipv6=0, tt->did_ifconfig_ipv6_setup=0
15/11/2019 17:24:36Fri Nov 15 16:24:36 2019 /sbin/ip link set dev tun0 up mtu 1500
15/11/2019 17:24:36Fri Nov 15 16:24:36 2019 /sbin/ip addr add dev tun0 local 10.82.171.1 peer 10.82.171.2
15/11/2019 17:24:36Fri Nov 15 16:24:36 2019 /sbin/ip route add 192.168.1.0/24 via 10.82.171.2
15/11/2019 17:24:36Fri Nov 15 16:24:36 2019 /sbin/ip route add 10.82.171.0/24 via 10.82.171.2
15/11/2019 17:24:36Fri Nov 15 16:24:36 2019 GID set to nogroup
15/11/2019 17:24:36Fri Nov 15 16:24:36 2019 UID set to nobody
15/11/2019 17:24:36Fri Nov 15 16:24:36 2019 Listening for incoming TCP connection on [undef]
15/11/2019 17:24:36Fri Nov 15 16:24:36 2019 TCPv4_SERVER link local (bound): [undef]
15/11/2019 17:24:36Fri Nov 15 16:24:36 2019 TCPv4_SERVER link remote: [undef]
15/11/2019 17:24:36Fri Nov 15 16:24:36 2019 MULTI: multi_init called, r=256 v=256
15/11/2019 17:24:36Fri Nov 15 16:24:36 2019 IFCONFIG POOL: base=10.82.171.4 size=62, ipv6=0
15/11/2019 17:24:36Fri Nov 15 16:24:36 2019 MULTI: TCP INIT maxclients=1024 maxevents=1028
15/11/2019 17:24:36Fri Nov 15 16:24:36 2019 Initialization Sequence Completed
And here's the debug log of the new image
+ set -e
+ cd /etc/openvpn
+ USER_ARGS=("${@}")
+ ARGS=()
+ addArg --config /etc/openvpn/openvpn.conf
+ local arg=--config
+ '[' 2 -ge 1 ']'
+ local val=/etc/openvpn/openvpn.conf
+ hasArg --config
+ local element
+ return 1
+ ARGS+=("${arg}")
+ '[' 2 -ge 1 ']'
+ ARGS+=("${val}")
+ source /etc/openvpn/ovpn_env.sh
++ declare -x OVPN_AUTH=
++ declare -x OVPN_CIPHER=
++ declare -x OVPN_CLIENT_TO_CLIENT=
++ declare -x OVPN_CN=vpn.sportcare.tech
++ declare -x OVPN_COMP_LZO=1
++ declare -x OVPN_DEFROUTE=0
++ declare -x OVPN_DEVICE=tun
++ declare -x OVPN_DEVICEN=0
++ declare -x OVPN_DISABLE_PUSH_BLOCK_DNS=0
++ declare -x OVPN_DNS=0
++ OVPN_DNS_SERVERS=([0]="8.8.8.8" [1]="8.8.4.4")
++ declare -x OVPN_DNS_SERVERS
++ declare -x OVPN_ENV=/etc/openvpn/ovpn_env.sh
++ OVPN_EXTRA_CLIENT_CONFIG=()
++ declare -x OVPN_EXTRA_CLIENT_CONFIG
++ OVPN_EXTRA_SERVER_CONFIG=()
++ declare -x OVPN_EXTRA_SERVER_CONFIG
++ declare -x OVPN_FRAGMENT=
++ declare -x 'OVPN_KEEPALIVE=10 60'
++ declare -x OVPN_MTU=
++ declare -x OVPN_NAT=1
++ declare -x OVPN_PORT=1194
++ declare -x OVPN_PROTO=tcp
++ OVPN_PUSH=([0]="route 192.168.5.0 255.255.255.0" [1]="route 192.168.5.0 255.255.255.0")
++ declare -x OVPN_PUSH
++ OVPN_ROUTES=([0]="192.168.1.0/24")
++ declare -x OVPN_ROUTES
++ declare -x OVPN_SERVER=172.16.33.0/24
++ declare -x OVPN_SERVER_URL=tcp://vpn.sportcare.tech
++ declare -x OVPN_TLS_CIPHER=
+ mkdir -p /dev/net
+ '[' '!' -c /dev/net/tun ']'
+ mknod /dev/net/tun c 10 200
+ '[' -d /etc/openvpn/ccd ']'
+ addArg --client-config-dir /etc/openvpn/ccd
+ local arg=--client-config-dir
+ '[' 2 -ge 1 ']'
+ local val=/etc/openvpn/ccd
+ hasArg --client-config-dir
+ local element
+ return 1
+ ARGS+=("${arg}")
+ '[' 2 -ge 1 ']'
+ ARGS+=("${val}")
+ '[' -z '' ']'
+ OVPN_NATDEVICE=eth0
+ '[' 0 '!=' 0 ']'
+ '[' 1 == 1 ']'
+ setupIptablesAndRouting
+ iptables -t nat -C POSTROUTING -s 172.16.33.0/24 -o eth0 -j MASQUERADE
iptables: No chain/target/match by that name.
+ iptables -t nat -A POSTROUTING -s 172.16.33.0/24 -o eth0 -j MASQUERADE
+ for i in "${OVPN_ROUTES[@]}"
+ iptables -t nat -C POSTROUTING -s 192.168.1.0/24 -o eth0 -j MASQUERADE
iptables: No chain/target/match by that name.
+ iptables -t nat -A POSTROUTING -s 192.168.1.0/24 -o eth0 -j MASQUERADE
+ '[' /etc/openvpn/pki/crl.pem -nt /etc/openvpn/crl.pem ']'
+ '[' -r /etc/openvpn/crl.pem ']'
+ addArg --crl-verify /etc/openvpn/crl.pem
+ local arg=--crl-verify
+ '[' 2 -ge 1 ']'
+ local val=/etc/openvpn/crl.pem
+ hasArg --crl-verify
+ local element
+ return 1
+ ARGS+=("${arg}")
+ '[' 2 -ge 1 ']'
+ ARGS+=("${val}")
+ ip -6 route show default
+ '[' 0 = 0 ']'
+ echo 'Enabling IPv6 Forwarding'
Enabling IPv6 Forwarding
+ sysctl -w net.ipv6.conf.all.disable_ipv6=0
sysctl: error setting key 'net.ipv6.conf.all.disable_ipv6': Read-only file system
+ echo 'Failed to enable IPv6 support'
Failed to enable IPv6 support
+ sysctl -w net.ipv6.conf.default.forwarding=1
sysctl: error setting key 'net.ipv6.conf.default.forwarding': Read-only file system
+ echo 'Failed to enable IPv6 Forwarding default'
Failed to enable IPv6 Forwarding default
+ sysctl -w net.ipv6.conf.all.forwarding=1
sysctl: error setting key 'net.ipv6.conf.all.forwarding': Read-only file system
+ echo 'Failed to enable IPv6 Forwarding'
Failed to enable IPv6 Forwarding
+ echo 'Running '\''openvpn --config' /etc/openvpn/openvpn.conf --client-config-dir /etc/openvpn/ccd --crl-verify '/etc/openvpn/crl.pem '\'''
Running 'openvpn --config /etc/openvpn/openvpn.conf --client-config-dir /etc/openvpn/ccd --crl-verify /etc/openvpn/crl.pem '
+ exec openvpn --config /etc/openvpn/openvpn.conf --client-config-dir /etc/openvpn/ccd --crl-verify /etc/openvpn/crl.pem
Fri Nov 15 16:27:14 2019 OpenVPN 2.4.7 x86_64-alpine-linux-musl [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [MH/PKTINFO] [AEAD] built on May 5 2019
Fri Nov 15 16:27:14 2019 library versions: OpenSSL 1.1.1c 28 May 2019, LZO 2.10
Fri Nov 15 16:27:14 2019 Diffie-Hellman initialized with 2048 bit key
Fri Nov 15 16:27:14 2019 Outgoing Control Channel Authentication: Using 160 bit message hash 'SHA1' for HMAC authentication
Fri Nov 15 16:27:14 2019 Incoming Control Channel Authentication: Using 160 bit message hash 'SHA1' for HMAC authentication
Fri Nov 15 16:27:14 2019 ROUTE_GATEWAY 169.254.1.1
Fri Nov 15 16:27:14 2019 TUN/TAP device tun0 opened
Fri Nov 15 16:27:14 2019 TUN/TAP TX queue length set to 100
Fri Nov 15 16:27:14 2019 /sbin/ip link set dev tun0 up mtu 1500
Fri Nov 15 16:27:14 2019 /sbin/ip addr add dev tun0 local 172.16.33.1 peer 172.16.33.2
Fri Nov 15 16:27:14 2019 /sbin/ip route add 192.168.1.0/24 via 172.16.33.2
Fri Nov 15 16:27:14 2019 /sbin/ip route add 172.16.33.0/24 via 172.16.33.2
Fri Nov 15 16:27:14 2019 Could not determine IPv4/IPv6 protocol. Using AF_INET
Fri Nov 15 16:27:14 2019 Socket Buffers: R=[87380->87380] S=[16384->16384]
Fri Nov 15 16:27:14 2019 Listening for incoming TCP connection on [AF_INET][undef]:1194
Fri Nov 15 16:27:14 2019 TCPv4_SERVER link local (bound): [AF_INET][undef]:1194
Fri Nov 15 16:27:14 2019 TCPv4_SERVER link remote: [AF_UNSPEC]
Fri Nov 15 16:27:14 2019 GID set to nogroup
Fri Nov 15 16:27:14 2019 UID set to nobody
Fri Nov 15 16:27:14 2019 MULTI: multi_init called, r=256 v=256
Fri Nov 15 16:27:14 2019 IFCONFIG POOL: base=172.16.33.4 size=62, ipv6=0
Fri Nov 15 16:27:14 2019 MULTI: TCP INIT maxclients=1024 maxevents=1028
Fri Nov 15 16:27:14 2019 Initialization Sequence Completed
I think the issue should be in this error in the new image
+ iptables -t nat -C POSTROUTING -s 172.16.33.0/24 -o eth0 -j MASQUERADE
iptables: No chain/target/match by that name.
+ iptables -t nat -A POSTROUTING -s 172.16.33.0/24 -o eth0 -j MASQUERADE
+ for i in "${OVPN_ROUTES[@]}"
+ iptables -t nat -C POSTROUTING -s 192.168.1.0/24 -o eth0 -j MASQUERADE
iptables: No chain/target/match by that name.
+ iptables -t nat -A POSTROUTING -s 192.168.1.0/24 -o eth0 -j MASQUERADE
However if I run iptables after the container is started I can see those rules
+ iptables -t nat -C POSTROUTING -s 172.16.33.0/24 -o eth0 -j MASQUERADE
iptables: No chain/target/match by that name.
+ iptables -t nat -A POSTROUTING -s 172.16.33.0/24 -o eth0 -j MASQUERADE
+ for i in "${OVPN_ROUTES[@]}"
+ iptables -t nat -C POSTROUTING -s 192.168.1.0/24 -o eth0 -j MASQUERADE
iptables: No chain/target/match by that name.
+ iptables -t nat -A POSTROUTING -s 192.168.1.0/24 -o eth0 -j MASQUERADE
Also another strange thing is in this lines:
# Old image
ROUTE_GATEWAY 192.168.5.1/255.255.255.0 IFACE=eth0 HWADDR=66:32:30:39:36:37
# New image
ROUTE_GATEWAY 169.254.1.1
Any hint?
I know this is a very old issue, but I ran into it as well, and I solved it by creating an init container that sets IP forwarding on the pod.
I know this is a very old issue, but I ran into it as well, and I solved it by creating an
initcontainer that sets IP forwarding on the pod.
Interesting, could you please share your setup?
Sure, I've built an OpenVPN 2.5 Docker image that is running as a deployment pod on Kubernetes 1.22 with a busybox init containter that sets net.ipv4.ip_forward=1. I've a configmap entry that pushes "route 10.11.1.0 255.255.255.0" which is my private subnet. It works both with or without redirect-gateway def1 (the difference being that my client's default GW would change, but I can still access my private subnet either way).