openvpn3-linux
openvpn3-linux copied to clipboard
OpenVPN
v15beta is not properly overriding DNS records Ubuntu 20.04
Hi,
We are trying to use this client to connect to openvpn cloud. We are using our custom dns nameservers on OpenVPN cloud configured through the DNS setting of OpenVPN cloud.
We can connect without any issue to the VPN but the DNS resolution is not working at all. We can ping/ssh/curl our different services via cli with the IP without any issue but DNS resolution is not working. It seems that systemd-resolve is using the public DNS and IP not matter what.
The thing to notice is that it is properly working on the beta13.
Did we miss some configuration steps to make it work properly ? You will find below the outputs from both beta15 and beta13
Output from beta15
systemd-resolv --status
Global
LLMNR setting: no
MulticastDNS setting: no
DNSOverTLS setting: no
DNSSEC setting: no
DNSSEC supported: no
DNSSEC NTA: 10.in-addr.arpa
16.172.in-addr.arpa
168.192.in-addr.arpa
17.172.in-addr.arpa
18.172.in-addr.arpa
19.172.in-addr.arpa
20.172.in-addr.arpa
21.172.in-addr.arpa
22.172.in-addr.arpa
23.172.in-addr.arpa
24.172.in-addr.arpa
25.172.in-addr.arpa
26.172.in-addr.arpa
27.172.in-addr.arpa
28.172.in-addr.arpa
29.172.in-addr.arpa
30.172.in-addr.arpa
31.172.in-addr.arpa
corp
d.f.ip6.arpa
home
internal
intranet
lan
local
private
test
Link 34 (tun0)
Current Scopes: DNS
DefaultRoute setting: yes
LLMNR setting: yes
MulticastDNS setting: no
DNSOverTLS setting: no
DNSSEC setting: no
DNSSEC supported: no
Current DNS Server: 100.96.1.33
DNS Servers: 100.96.1.33
DNS Domain: ~.
Link 3 (wlo1)
Current Scopes: DNS
DefaultRoute setting: yes
LLMNR setting: yes
MulticastDNS setting: no
DNSOverTLS setting: no
DNSSEC setting: no
DNSSEC supported: no
Current DNS Server: 192.168.43.137
DNS Servers: 2a04:cec0:1122:8deb::54
192.168.43.137
DNS Domain: ~.
Link 2 (eno2)
Current Scopes: none
DefaultRoute setting: no
LLMNR setting: yes
MulticastDNS setting: no
DNSOverTLS setting: no
DNSSEC setting: no
DNSSEC supported: no
cat /etc/resolv.conf
# This file is managed by man:systemd-resolved(8). Do not edit.
#
# This is a dynamic resolv.conf file for connecting local clients to the
# internal DNS stub resolver of systemd-resolved. This file lists all
# configured search domains.
#
# Run "resolvectl status" to see details about the uplink DNS servers
# currently in use.
#
# Third party programs must not access this file directly, but only through the
# symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a different way,
# replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.
nameserver 127.0.0.53
options edns0 trust-ad
Output from Beta13
systemd-resolve --status
Global
Protocols: -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
resolv.conf mode: foreign
Current DNS Server: 100.96.1.33
DNS Servers: 100.96.1.33 127.0.1.1
DNS Domain: net
Link 2 (eno2)
Current Scopes: none
Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Link 3 (enx64c901d792c8)
Current Scopes: DNS
Protocols: +DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: 192.168.1.1
DNS Servers: 192.168.1.1
DNS Domain: net
Link 4 (wlo1)
Current Scopes: none
Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Link 5 (docker0)
Current Scopes: none
Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Link 7 (tun1)
Current Scopes: none
Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
cat /etc/resolv.conf
#
# Generated by OpenVPN 3 Linux (NetCfg::DNS::ResolvConfFile)
# Last updated: 2021-07-23 14:02:19
#
search net
# OpenVPN defined name servers
nameserver 100.96.1.33
# System defined name servers
nameserver 127.0.1.1
# Other system settings
options edns0 trust-ad
Same issue on Pop!_OS 20.04 LTS
v13 systemd-resolve --status
Link 4 (tun0)
Current Scopes: none
DefaultRoute setting: no
LLMNR setting: yes
MulticastDNS setting: no
DNSOverTLS setting: no
DNSSEC setting: no
DNSSEC supported: no
v13 /etc/resolv.conf
# OpenVPN defined name servers
nameserver 192.168.16.254
# System defined name servers
nameserver 127.0.0.53
v15 systemd-resolve --status
Link 6 (tun0)
Current Scopes: DNS
DefaultRoute setting: yes
LLMNR setting: yes
MulticastDNS setting: no
DNSOverTLS setting: no
DNSSEC setting: no
DNSSEC supported: no
Current DNS Server: 192.168.16.254
DNS Servers: 192.168.16.254
DNS Domain: ~.
v15 /etc/resolv.conf
# System defined name servers
nameserver 127.0.0.53
That DNS Domain is not correct. It is the same as the default DNS config in systemd-resolv which causes it to use one of them at random.
OpenVPN As version 2.8.7 "DNS zones" empty since we want to use our DNS for everything. "Default domain suffix" filled, but that is ignored.
*edit after reading how systemd-resolved works:
- we have a public internet presence at *.example.com (wildcard dns)
- we have a private network at *.private.example.com (only resolvable by our own dns server over the vpn)
on v13: only the vpn-defined DNS server is used, both public and private urls resolve. on v15: both the standard as wel as the vpn-defined DNS servers are used and the 1st non-failing result is returned. Private urls only resolve if the vpn DNS returns a success before the standard dns does.
Hello!
Is it possible to force openvpn change resolv.conf as before?
Thank you!
What does the hosts: line in your /etc/nsswitch.conf file contain?
In v14_beta/v15_beta openvpn3-llinux switched over to use systemd-resolved integration by default. But if the NSS configuration is incorrect, it will query the wrong resolver. On my RHEL-8 box, it looks like this:
hosts: files mdns4_minimal [NOTFOUND=return] resolve dns myhostname
You must ensure resolve (aka systemd-resolved) is enslisted before dns (aka /etc/resolv.conf)
To quickly switch back to modifying /etc/resolv.conf you can modify /usr/share/dbus-1/system-services/net.openvpn.v3.netcfg.service and replace the --systemd-resolved argument with --resolv-conf /etc/resolv.conf. But that is just a hackish workaround and it will be reset on the next upgrade.
Well, Ubuntu 20.04 is completely different from RHEL here, as I can see.
/etc/nsswitch.conf changed nothing here, it contains hosts: files dns in my installation and adding resolve has no effect.
But just found this issue https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1774632 changed resolv.conf to symlink as suggested. Looks like it works...
After connection using openvpn2 I see my internal nameserver in /run/systemd/resolve/resolv.conf but it is not removed after disconnect...
So I changed everything back to resolv.conf method...
Thank you!
Okay, so the launchpad OP is actually configuring systemd-resolved in a different way, not fully utilizing the possibilities with systemd-resolved. To be honest, I have no idea why Ubuntu is not making use of the resolve support via nsswitch.conf, or why that is not working for you.
It is well spent time reading through the systemd-resolved(8) man page.
So when just re-linking /etc/resolv.conf to /run/systemd/resolved/resolv.conf, depending on the priority of devices systemd-resolved has, it may not put the VPN provided DNS servers first. On my RHEL system, I see the DNS server provided via the ethernet interface enlisted first and then at the end comes the VPN ones in that file. But using the hosts line you've seen above together with /etc/resolv.conf being a symlink to /run/systemd/resolve/stub-resolv.conf gives the expected behavior on for me.
Is this box running in some kind of VPS service? Or is it bare metal? I do know that several of the VPS providers and cloud providers may have default images which deviates from the way stock Ubuntu install is configuring the host. Several of them might even ignore the fact that the distro has moved towards using systemd-resolved for DNS resolving; which makes this quite more complicated.
/run/systemd/resolve/stub-resolv.conf is not changed in Ubuntu 20.04, nameserver is only added to /run/systemd/resolve/resolv.conf , and, as I wrote before, openvpn nameserver is not removed from /run/systemd/resolve/resolv.conf after disconnect.
my system is bare metal, this is home desktop, but, as I can see on fresh 20.04 install in VM there is no resolve in nsswitch.conf.
btw, what is interesting here - reverse zone resolution works OK. I looked into tcpdump and I see that Ubuntu asks both nameservers- system ( let's call it this name) and openvpn, but prefer answer from system. So, really, systemd-resolved is broken in Ubuntu , at least by default, so we need resolv.conf support...
Thank you!
That's really interesting. Also because I have reports now from other channels that DNS resolution is finally working as expected on 20.04 - where modifying resolv.conf directly caused issues. So 20.04 is clearly a challenging case.
I'm a bit at loss here now, how to proceed, as the "group" of users which was unhappy with this change are now happy, and vice versa. I'll at least try to improve how to switch between resolv.conf and systemd-resolved.
In a nutshell: previously: dns servers are queried sequentially. now: all dns servers are queried in parallel. In both cases the first non-fail answer is used.
So if your regular DNS responds with a NXDOMAIN answer before the VPN-specified DNS server responds with an answer you're out of luck.
This is the default behaviour without "DNS Resolution Zones" (systemd-resolve --status shows the '~.' domain at both normal and vpn interface)
You can configure these "DNS Resolution Zones" in your OpenVPN server to force the vpn dns server to be used for certain domains.
You can not ensure the vpn dns is always used first before any other dns.
And this is the behaviour that people depended upon and which has now changed.
Ubuntu 20.04 indeed does not use resolve in nsswitch.
It uses /etc/resolv.conf and tries the nameservers listed there sequentially.
The last (and now the only) entry is 127.0.0.53 which is the systemd-resolve daemon.
That looks at the nameservers and domains specified for the interfaces and uses all that match the requested domain ('~.' indicates the nameserver should be tried for all domains.
@Kender2 Thanks for this clarification. This is gold, as it also gives a reasonable explanation why some users are more happy while some not with this change - on many setups, the VPN response is fast enough to not cause any issues - or that the DNS server otherwise used is getting blocked due to routes pushed out by the VPN server, which either slows down or makes that DNS inaccessible - while the VPN provided DNS server is available.
I've been pondering on the next systemd-resolved improvement, to get closer to the old behavior of "use only VPN DNS servers". Should we attempt to loop through all the interfaces and remove the ~. entry for all interfaces except the VPN (most likely enabling through some configuration flag)? (And naturally restore it again when the VPN session closes) ... Any thoughts on this?
That could work I think. You'd have to sure to only do that when not using "split dns". The ~. entry for the vpn interface only appears when no domain is filled in the OpenVPN server.
It'd also remove the fallback behaviour where the next dns server is tried when the vpn one doesn't respond (or responds with NoAnswer due to dns rebinding protection or such)
You'd better make a sheet/matrix with all the possible combinations of configurations and resulting behaviours first.
Quite a puzzle.
I'd like to add that v16 resolving is still broken on ubuntu 20.04 , I see no difference from v15...
Thanks for testing that. We still have some work on improving the DNS configuration. What should be easier with v16_beta is to revert back to the "old way" of manipulating /etc/resolv.conf. This should only be considered a quick workaround and not an ideal final solution.
To switch back, run this command: # openvpn3-admin netcfg-service --config-set resolv-conf /etc/resolv.conf # openvpn3-admin netcfg-service --config-set systemd-resolved false
You will need to stop the openvpn3-service-netcfg service before these changes will be activated. Currently the only way to restart this service is to kill -INT the process, reboot the system or disconnect all VPN sessions and wait 10 minutes and verify that the process has stopped running.
Same on regolith/focal. openvpn3 v17_beta. Trying the workaround above works sometimes but it seems a bit random (or volatile at least).
Thanks for testing that. We still have some work on improving the DNS configuration. What should be easier with v16_beta is to revert back to the "old way" of manipulating
/etc/resolv.conf. This should only be considered a quick workaround and not an ideal final solution.To switch back, run this command: # openvpn3-admin netcfg-service --config-set resolv-conf /etc/resolv-conf # openvpn3-admin netcfg-service --config-set systemd-resolved false
You will need to stop the
openvpn3-service-netcfgservice before these changes will be activated. Currently the only way to restart this service is tokill -INTthe process, reboot the system or disconnect all VPN sessions and wait 10 minutes and verify that the process has stopped running.
Is that a typo in the filename /etc/resolv-conf? Shoudn't it be /etc/resolv.conf?
I don't really understand what I'm doing but from the output of running it, there seems to be a config file. There was none on my machine but there is nothing indicating failure.
$ openvpn3-admin netcfg-service --config-set resolv-conf /etc/resolv.conf
Loading configuration file: /var/lib/openvpn3/netcfg.json
Configuration file updated. Changes will be activated next time openvpn3-service-netcfg restarts
$ openvpn3-admin netcfg-service --config-set systemd-resolved false
Loading configuration file: /var/lib/openvpn3/netcfg.json
Configuration file updated. Changes will be activated next time openvpn3-service-netcfg restarts
There is still no config file
$ cat /var/lib/openvpn3/netcfg.json
cat: /var/lib/openvpn3/netcfg.json: No such file or directory
If I create the file manually, the config changes:
$ sudo touch /var/lib/openvpn3/netcfg.json
$ sudo chown "$USERNAME" /var/lib/openvpn3/netcfg.json
$ openvpn3-admin netcfg-service --config-set resolv-conf /etc/resolv.conf
Loading configuration file: /var/lib/openvpn3/netcfg.json
Configuration file updated. Changes will be activated next time openvpn3-service-netcfg restarts
$ cat /var/lib/openvpn3/netcfg.json
{
// Option --resolv-conf :: resolv-conf file
"resolv_conf_file" : "/etc/resolv.conf"
}
Now here comes the kicker: The config is no longer allowed when running the second line(!):
$ openvpn3-admin netcfg-service --config-set systemd-resolved false
Loading configuration file: /var/lib/openvpn3/netcfg.json
Configuration NOT changed due to the following error:
*** These options cannot be combined: resolv-conf, systemd-resolved
Is that a typo in the filename
/etc/resolv-conf? Shoudn't it be/etc/resolv.conf?
Oh, yes! That's a typo. I've fixed it. Thanks!
I don't really understand what I'm doing but from the output of running it, there seems to be a config file. There was none on my machine but there is nothing indicating failure.
$ openvpn3-admin netcfg-service --config-set resolv-conf /etc/resolv.conf Loading configuration file: /var/lib/openvpn3/netcfg.json Configuration file updated. Changes will be activated next time openvpn3-service-netcfg restarts
Okay, that's unfortunate. It should have been an error here. openvpn3-admin is intended to be run as root. I'll file that as a bug internally.
If I create the file manually, the config changes:
$ sudo touch /var/lib/openvpn3/netcfg.json $ sudo chown "$USERNAME" /var/lib/openvpn3/netcfg.json
This step should not be needed when running openvpn3-admin as root. The openvpn user normally needs to own these files.
$ openvpn3-admin netcfg-service --config-set resolv-conf /etc/resolv.conf Loading configuration file: /var/lib/openvpn3/netcfg.json Configuration file updated. Changes will be activated next time openvpn3-service-netcfg restarts $ cat /var/lib/openvpn3/netcfg.json { // Option --resolv-conf :: resolv-conf file "resolv_conf_file" : "/etc/resolv.conf" }
Now here comes the kicker: The config is no longer allowed when running the second line(!):$ openvpn3-admin netcfg-service --config-set systemd-resolved false Loading configuration file: /var/lib/openvpn3/netcfg.json Configuration NOT changed due to the following error:
*** These options cannot be combined: resolv-conf, systemd-resolved
Yikes ... this was intended to work. I'll double check this. This will also be filed as a bug internally.
Thanks again for thorough testing and documenting.
I figured out that I needed to add sudo, after a couple of hours of debugging. Some more feedback from openvpn3-admin would have been nice :dancers:. Just so summarize the two things there:
- Warn if not running as root
- Double check that the config file exists and was actually written. The log saying
Configuration file updatedis very much false.
tl;dr: Activate the workaround by running
# disconnect
openvpn3 session-manage --path $(openvpn3 sessions-list | head -n 2 | tail -n 1 | cut -c 15-) --disconnect
sudo openvpn3-admin netcfg-service --config-set resolv-conf /etc/resolv.conf
sudo openvpn3-admin netcfg-service --config-set systemd-resolved false # Not working for me
# Make sure the config is reloaded. openvpn3-service-netcfg must be killed.
sudo killall -INT openvpn3-service-netcfg
In regards to the error related to this command line:
sudo openvpn3-admin netcfg-service --config-set systemd-resolved false # Not working for me
This is actually not a bug. It's me who confused it with some prior implementations which I dropped. Instead it should say:
sudo openvpn3-admin netcfg-service --config-unset systemd-resolved
And it should be called first, before the --config-set resolv-conf command line.
I don't really understand what I'm doing but from the output of running it, there seems to be a config file. There was none on my machine but there is nothing indicating failure.
$ openvpn3-admin netcfg-service --config-set resolv-conf /etc/resolv.conf Loading configuration file: /var/lib/openvpn3/netcfg.json Configuration file updated. Changes will be activated next time openvpn3-service-netcfg restartsOkay, that's unfortunate. It should have been an error here.
openvpn3-adminis intended to be run asroot. I'll file that as a bug internally.
This should now be fixed with commit c40218df43c8e652fedfa70304eae797b305e780. The output now is still not optimal, but at least gives a pretty good idea:
$ openvpn3-admin netcfg-service --config-set resolv-conf /tmp/test
Loading configuration file: /var/lib/openvpn3/netcfg.json
** ERROR ** Configuration file error in /var/lib/openvpn3/netcfg.json: Error saving the configuration file
I was having this same issue, I'm using Ubuntu 20.04, with OpenVPN 3/Linux v18_beta (openvpn3).
I followed the instructions above (thanks johantiden and dsommers) and resolved it by:
user@hostname:~$ openvpn3 session-manage --path $(openvpn3 sessions-list | head -n 2 | tail -n 1 | cut -c 15-) --disconnect
Initiated session shutdown.
Connection statistics:
BYTES_IN...............119344058
BYTES_OUT...............85034916
PACKETS_IN................350732
PACKETS_OUT...............651178
TUN_BYTES_IN............66570523
TUN_BYTES_OUT..........104480638
TUN_PACKETS_IN............609700
TUN_PACKETS_OUT...........470655
KEEPALIVE_TIMEOUT..............2
N_RECONNECT...................10
user@hostname:~$ sudo openvpn3-admin netcfg-service --config-unset systemd-resolved
Loading configuration file: /var/lib/openvpn3/netcfg.json
user@hostname:~$ sudo openvpn3-admin netcfg-service --config-set resolv-conf /etc/resolv.conf
Loading configuration file: /var/lib/openvpn3/netcfg.json
Configuration file updated. Changes will be activated next time openvpn3-service-netcfg restarts
user@hostname:~$ sudo killall -INT openvpn3-service-netcfg
After that I connected the usual way and DNS behaved as expected (finally!). The proper behavior persists even after disconnect and reconnect, as well as after reboot.
I guess someday I will see if 22.04 handles this better.
I was in the process of getting my machine set up, with a clean install (22.04), and ran into the same issue... @walterbray your steps worked well for me. Really learned from the discussion too as I'm still new to Linux. Thanks all! Just dropping in to say that this seems to affect 22.04 as well, or at least it did in my case.
