Improve DNS handling?

[DimitriPapadopoulos]

Rationale

Setting DNS parameters has been raising different issues:

• race condition between openfortivpn itself and pppd in earlier versions of openfortivpn prior to #486 and version 1.12.0
• race condition between openfortivpn and NetworkManager (see for example #590)
• mechanisms for setting DNS might be available at build-time but not run-time, or at run-time but not build-time (for example resolvectl from systemd is often available at run-time on Fedora machines but is not available at build-time by default)
• mechanisms for setting DNS might be available but not working (for example resolvconf from systemd does not work as expected on Fedora as seen in #555)

Especially Fedora and CentOS have been challenging so far. Short of interacting with NetworkManager directly when NetworkManager is installed and used, I suspect the official way to modify DNS parameters on CentOS >= 8 and Fedora is running resolvectl which has been part of the systemd package since CentOS >= 8 (not CentOS 7 which is probably why busctl is used in #615).

Proposal for setting DNS parameters

Is it worth trying to further improve the DNS configuration mechanism when --set-dns=1? Perhaps using the following mechanisms and other ones you can think of?

1. If NetworkManager is installed and in use (how to detect that?) interact with NetworkManager directly? Perhaps through nmcli?
2. If resolvectl from systemd is available run it to modify DNS parameters.
3. If resolvconf is available and works properly (either openresolv or resolvconf but not resolvectl), run it to modify DNS parameters.
4. Fall back on modifying /etc/resolv.conf directly.

NetworkManager is a special case: I am not sure about supporting NetworkManager because NetworkManager already supports openfortivpn through NetworkManager-fortisslvpn. It's the other way round.

Documentation

Man pages:

• nmcli(1) from NetworkManager
• resolvectl(1) from systemd
• resolvconf(8) from Debian
• resolvconf(8) from openresolv

Hopefully we can get some info from this bug report: https://bugzilla.redhat.com/show_bug.cgi?id=1815605

[DimitriPapadopoulos]

Additionally I believe we need to always support all of resolvectl, resolvconf and modifying /etc/resolv.conf directly. openfortivpn should decide at run-time and not build-time. At least this looks like the only way to support snaps because DNS management is distribution-specific on Linux while snaps are distribution-neutral so they have to support all distributions. See https://github.com/adrienverge/openfortivpn/pull/597#issuecomment-605649182.

[DimitriPapadopoulos]

I have tried to create a list of resolvectl and resolvconf executables on different Linux distributions based on my own experience and pkgs.org:

executables (or lack of)	packages
ALT Linux Sisyphus
shell function resolvectl?	systemd
/sbin/resolvconf	openresolv
Arch Linux
/usr/bin/resolvectl	systemd
/usr/bin/resolvconf	resolvconf
/usr/bin/resolvconf	openresolv
CentOS 8
/usr/bin/resolvectl	systemd
/usr/sbin/resolvconf	systemd
CentOS 7
	systemd
CentOS 6
/sbin/resolvconf	openresolv
Debian 10
/usr/bin/resolvectl	systemd
/sbin/resolvconf	resolvconf
/sbin/resolvconf	openresolv
Debian 9
	systemd
/sbin/resolvconf	resolvconf
/sbin/resolvconf	openresolv
Debian 8
	systemd
/sbin/resolvconf	resolvconf
Fedora 32
/usr/bin/resolvectl	systemd
/usr/sbin/resolvconf	systemd
Fedora 31
/usr/bin/resolvectl	systemd
/usr/sbin/resolvconf	systemd
Fedora 30
/usr/bin/resolvectl	systemd
/usr/sbin/resolvconf	systemd
FreeBSD 13
	systemd
/usr/local/sbin/resolvconf	openresolv
OpenMandriva 4.1
/usr/bin/resolvectl	systemd
/sbin/resolvconf	systemd
OpenMandriva 3.0
	systemd
OpenSuSE Ports Leap 15.2
	systemd
/usr/sbin/resolvconf	resolvconf
OpenSuSE Ports Leap 15.1
	systemd
/usr/sbin/resolvconf	resolvconf
Slackware 14.2
/usr/sbin/resolvconf	openresolv
Slackware 14.1
/usr/sbin/resolvconf	openresolv
PCLinuxOS
/sbin/resolvconf	resolvconf
Solus
/usr/bin/resolvectl	systemd
/usr/sbin/resolvconf	systemd
Ubuntu 20.04
/usr/bin/resolvectl	systemd
/sbin/resolvconf	resolvconf
/sbin/resolvconf	openresolv
Ubuntu 18.04
	systemd
/sbin/resolvconf	resolvconf
/sbin/resolvconf	openresolv
Ubuntu 16.04
	systemd
/sbin/resolvconf	resolvconf
/sbin/resolvconf	openresolv

Most notably resolvectl has been gaining momentum on most distributions, especially Debian 10 and Ubuntu 20.04. Hence if --set-dns=1 we should by order of preference:

1. use resolvectl from systemd if available,
2. else use resolvconf if available, the one from resolvconf or openresolv and certainly not the broken resolvectl alias from systemd,
3. else modify /etc/resolv.conf directly.

[mrbaseman]

Supporting resolvectl surely is something which we should integrate somehow in the near future.

I'm a bit hesitating about the movement of the decision from configure time to the runtime. Programs running with elevated privileges should call other programs with an absolute path. If this decision can not be detected at configure time, we may end up with checking several possibilities at runtime and expectations e.g. about how trustworthy directories below /usr or /usr/local are, may also depend on the distribution.

So, shall we check a couple of locations which we think are safe at runtime? Or shall we rely on a mechanism of the OS to locate the right executable (use $PATH and sanitize it somehow), anything else? Whatever we chose, it should also be portable.

One could argue, openfortivpn needs to run as root anyway, so the user who is allowed to use it must be trustworthy in some sense, but I'm sure some admins provide their users with sudo privileges, the privilege to control network manager, or a systemd service that opens the vpn tunnel on request by the otherwise less trusted employee. I feel a bit uncomfortable with the idea of putting too much automatism to runtime.

[DimitriPapadopoulos]

I absolutely agree, we should not let openfortivpn search the PATH or anything else to find a way to modify DNS parameters. Paths need to be hardcoded. That's why I have researched the location of the executables for various distributions. Therefore by default we should attempt to run only the following executables in directories controlled by the system:

1. /usr/bin/resolvectl is the standard location for resolvectl from systemd across distributions.
2. /sbin/resolvconf and /usr/sbin/resolvconf look like standard locations for resolvconf/openresolv across distributions. Perhaps add /usr/bin/resolvconf for Arch Linux and /usr/local/sbin/resolvconf for FreeBSD, but the last one can and should be configured at build-time.

Note that the broken systemd alias /sbin/resolvconf or /usr/sbin/resolvconf seems to be "hidden" by /usr/bin/resolvectl in all cases, provided we attempt to run resolvectl before resolvconf.

We should also allow for additional/alternative paths to these executables at build-time, for obscure distributions that use different paths or end-users who build & install themselves in different locations.

Finally we should examine the exit status of the executables to catch errors. Will have to experiment with exit status values a little bit.

[DimitriPapadopoulos]

A relevant pull request has just appeared: #615

@da-x Can you please help understand the pull request and comment with respect to other solutions above?

• If I understand the pull request correctly, you're calling busctl instead of resolvectl. Can you comment on that? Perhaps because busctl is available on more distributions than resolvectl (Ubuntu 16.04 and 18.04, Debian 8 and 9, CentOS 7)?
• We already link with systemd libraries. I'm wondering whether we could call systemd functions directly from C instead of running the external script.
• I've been struggling to find the "right way" to modify DNS settings from a program such as openfortivpn. Can you confirm that systemd is the way to go on systems with systemd (so all systems listed above except CentOS 6 and SlackWare), whether using resolvectl, busctl or direct systemd function calls from C? Can you point me to some external publication to support that (I'm convinced but I would like to read a little bit about it)?
• All recent distributions that ship some form of resolvconf also ship systemd, see table above. If systemd is the way to go, I suspect the suggestion to support resolvconf (#486) was a bad idea. Your opinion?

[DimitriPapadopoulos]

Here are the services provided by systemd-resolved: https://www.freedesktop.org/wiki/Software/systemd/resolved/

@da-x You seem to be calling methods SetLinkDNS, SetLinkDomainsand also SetLinkDNSSEC. Is the last one required? I'm not very familiar with DNSSEC

[da-x]

@DimitriPapadopoulos this bash script code is based on code from https://github.com/jonathanio/update-systemd-resolved, which does similar stuff but for OpenVPN. You can check with them. Now, there's also a hidden assumption that systemd-resolved is enabled on the system, and that /etc/nsswitch.conf is configured for it, which I think is not yet true by default for some distribution versions. I, for one, use it, because I have a home DNS server and a work DNS server, and resolving via system-resolved works well when I try to resolve hostnames from either networks.

[DimitriPapadopoulos]

@da-x Right, on Fedora systemd-resolved is not enabled by default as far as I can tell. Is your script able to detect that?

[da-x]

@DimitriPapadopoulos No, there's an assumption that it's enabled and /etc/nsswitch.conf is configured for it.

[DimitriPapadopoulos]

@da-x Then I suspect it might be much better to rewrite the script directly in C (since openfortivpn is already linked with systemd). Then:

1. Hopefully the C systemd function calls will return an error status if systemd-resolved is not enabled. By the way, if the C function calls fail, I suspect running resolvectl is useless since systemd-resolved is the only supported backend, isn't it?
2. If systemd is not linked in, or the C function calls fail, then we can fall back on resolvconf on systems that have the script (the real resolvconf/openresolve resolvconf script and not the systemd resolveclt alias).
3. If all else fail we fall back on directly modifying /etc/resolv.conf (for example on CentOS 6).

[da-x]

If systemd-resolved is not enabled, then /etc/nsswitch.conf configuration can be made to fallback to resolveconf. Order matters in how backends are specified in nsswitch.conf. In other words, it depends on whether you have:

hosts: files dns resolve

Or:

hosts: files resolve dns

Where files mean direct resolve from /etc/hosts, resolve is for systemd-resolved, and dns is for /etc/resolv.conf.

Also, I used the script as a stop-gap. I'm not sure I can dedicate the time now to implement it in C. So feel free to close that PR for now, or go about implementing it in C.

[DimitriPapadopoulos]

On Ubuntu 18.04:

hosts:          files mdns4_minimal [NOTFOUND=return] dns myhostname

On Fedora 31:

hosts:      files mdns4_minimal [NOTFOUND=return] dns myhostname

The order you are referring to is for DNS resolution which is not really relevant here. What we are after is the order of the tools to try to modify the parameters of DNS resolution. I believe these are two different things.

[DimitriPapadopoulos]

Interesting link: https://systemd-devel.freedesktop.narkive.com/ghlVQMfi/sd-bus-example-code-for-setlinkdns

[da-x]

Important to note is that if resolve does not appear in /etc/hosts, then most programs, which do not resolve directly via systemd-resolved will not benefit from the update, even if the service is installed and enabled.

[DimitriPapadopoulos]

It's getting more and more complicated:

• Different subsystems co-exist and can be used for name resolution. Of interest are the C library and the Name Service Switch (NSS), and systemd.
• Updating one of the subsystems doesn't necessarily update other subsystems. For example on Fedora 31 updating systemd has no effect on nslookup which is using the C library and NSS. Therefore we might have to update one subsystem and the others, not one subsystem or the other.
• Yet it really depends not on the current Linux distribution but also one the local setup of the distribution. For example it is expected that systemd will be the default name resolution subsystem in Fedora 33 - but only the default! Another example is Fedora 31 where systemd-resolved is installed but is not enabled by default.
• It will probably remain quite complicated to infer which name resolution subsystem needs updating and which doesn't for years to come.

Still investigating...

[DimitriPapadopoulos]

Note that resolvconf (either from openresolv or resolvconf) does not seem to be available by default on Ubuntu 20.04.

[DimitriPapadopoulos]

This ArchWiki OpenVPN page is quite interesting:

For Linux, the OpenVPN client can receive DNS host information from the server, but the client expects an external command to act on this information. No such commands are configured by default. They must be specified with the up and down config options. There are a few alternatives for what scripts to use, but none are officially recognised by OpenVPN [...]

[da-x]

This ArchWiki OpenVPN page is quite interesting:

For Linux, the OpenVPN client can receive DNS host information from the server, but the client expects an external command to act on this information. No such commands are configured by default. They must be specified with the up and down config options. There are a few alternatives for what scripts to use, but none are officially recognised by OpenVPN [...]

That's why I implemented that same approach for openfortivpn - running an external user-configured script. Looking at this I got the feel that the variety of DNS configurations and methods may be too wide for openfortivpn to guess what's preferred for the current system, especially if you want to support old systems. There isn't yet a widespread acceptable default across all distributions.

[DimitriPapadopoulos]

A few links of interest:

• How exactly are NetworkManager, networkd, netplan, ifupdown2, and iproute2 interacting?
• Ubuntu Network Manager: Enabling and disabling NetworkManager on Ubuntu
• Netplan
• Netplan Design
• Have a Plan for Netplan

[MauroMombelli]

hi, without going too deep, seems like systemd-resoved is quite common, so it would make sense to start implement it.
For NetworkManager should be relatively easy to check if the service is running and ask the user to use the proper plugin instead.
Also the interface could show the list of dns setup so if someone has a special need he can easily find the information and apply; this is also why i like the idea of the external script to set it up, it could be very nice if it look in the user default directory ($XDG_CONFIG_HOME)for a script with the name of the conf, so multiple script can coexist.

This would make extremely easy to share script with friend and colleague

[DimitriPapadopoulos]

Also the interface could show the list of dns setup so if someone has a special need he can easily find the information and apply; this is also why i like the idea of the external script to set it up, it could be very nice if it look in the user default directory ($XDG_CONFIG_HOME)for a script with the name of the conf, so multiple script can coexist.

On the other hand such scripts usually need to run with root privileges. It seems wrong to allow that for random scripts.

[MauroMombelli]

yes permission should be dropped to user if run from user folder, is a problem of the admin to give right permission to user (i believe some systemd stuff will ask for password if a GUI is running, at least systemctl does)

[DimitriPapadopoulos]

Also I'm not certain I understand systemd-resolved, perhaps you can shed some light:

• systemd-resolved is a subsystem that is often installed, but not necessarily enabled. For example on Fedora 31 workstations it is not enabled by default. How do we detect whether systemd-resolved is installed/enabled? Do we just send a systemd query and wait for a reply?
• systemd-resolved might be enabled but have no effect on the usual Glibc/NSS. For example that's the case on Fedora 31: after enabling systemd-resolved, nslookup and most programs based on Glibc are not affected by whatever changes are applied to systemd-resolved. It won't be until Fedora 33 that Glibc will switch from nss-dns to nss-resolve. Perhaps systemd-resolved is not the proper subsystem to talk to after all.
• Instead should we perhaps talk to NetworkManager or systemd-netword directly?

[MauroMombelli]

as @da-x pointed out, there are a ton of different configuration just by looking at standard distro, and some people on some machine may have ad-hoc network setup

Do we just send a systemd query and wait for a reply?

yes, as it is done by an external script is just as easy as

systemctl is-active --quiet systemd-resolved && echo "systemd-resolved is running"

the same can be done for NetworkManager

might be enabled but have no effect on the usual Glibc/NSS

if it is enabled we just assume is the main way to deal with it, as if we see NetworkManager we assume that is the main system. Again, if user has some weir setup he can just adjust its script, if is a common setup the script can be patched in future version, but user can easily override the script until release.

Instead should we perhaps talk to NetworkManager or systemd-netword directly?

IMHO we check for NetworkManager, if not enabled with systemd-resolved, if not enabled with /etc/hosts (add whatever you feel necessary) I have no idea why you talk about systemd-netword, afaik its functionality are orthogonal to resolved

[DimitriPapadopoulos]

Thank you. This helped me a lot, especially the explanation that a computer with systemd-resolved "enabled" but without effect on Glibc is probably ill-configured.

I have no idea why you talk about systemd-netword, afaik its functionality are orthogonal to resolved

I have no experience with systemd-networkd, so bear with me If I don't make sense. My understanding is that it is "equivalent" to NetworkManager:

• Ubuntu workstations have been using NetworkManager for quite some time. I understand NetworkManager oversees all network configuration and changes and talks to systemd-resolved and/or NSS for everything related to name resolution.
• Ubuntu servers on the other hand have been running systemd-networkd (at least since 18.04). So I thought the play a similar role. But perhaps the architecture is different and systemd-networkd is not "above" systemd-resolved as seems to be the case with NetworkManager. Also we not only need to change name resolution (with split DNS support in recent FortiOS versions) but also routing, hence the need for networkd-systemd.

I have been trying to read about NetworkManager, systemd-networkd and systemd-resolved, but my time is unfortunately limited. Do you know of any good introductory texts on:

• network configuration management on Linux systems, with NetworkdManager, systemd-networkd or other equivalent subsystems,
• name resolution configuration, probably in relation with the above?

So far I have found the ArchLinux documentation quite useful, for example:

• systemd-networkd
• systemd-resolved
• Domain name resolution

[MauroMombelli]

I am not super expert, but I had played around with both of them.

If NetworkManager is the frontend, systemd is the backend: NM may very well use ssytemd, plus more stull like wpa_supplicant an similar. I believe NM is born for the GUI interface, and being ported in different distro with different set of underling program for the network, it has gained compatibility with a vast set of backbend programs.

This is why if NM is present and active, he should have priority, as he is probably controlling the rest.

When he is not present, then we start to have the fragmented system, and the feasibility of a script that the user (or the superuser) can change to adapt to special case (lets imagine a machine running different container that need the VPN connection, or maybe only some, so you want to set up a special set of route when the tunnel is open!)

[DimitriPapadopoulos]

Thank you very much, it really helps. Attempting to call NetworkManager first makes sense. Therefore, unless the caller takes over routing and name resolution management, openfortivpn should configure name resolution by attempting to:

1. call NetworkManager if enabled,
2. call systemd-resolved if enabled,
3. call resolvconf if it is the proper resolvconf or openresolv script and not the broken bundled with systemd,
4. if all else fails modify /etc/resolv.conf directly.

It should also modify routing by attempting to:

1. call NetworkManager if enabled,
2. modify routes the usual way (not sure systemd-networkd can help here because of its rather static nature where network configuration depends on configuration files read at initialization).

I believe we might skip step 1 in both cases, at least in the short term, as NetworkManager-fortisslvpn takes care of these steps for us - but we might want to add them if we want openfortivpn to be able to work standalone.

@MauroMombelli and @da-x Does this make sense?

Finally note again that the modus operandi for name resolution will fail if systemd-resolved is enabled but not properly configured. In particular the three interfaces for network name resolution should never return inconsistent results, which is what I had experienced on Fedora 31 after merely running systemctl start systemd-resolved.service:

• The native, fully-featured API systemd-resolved exposes on the bus. See the API Documentation for details. Usage of this API is generally recommended to clients as it is asynchronous and fully featured (for example, properly returns DNSSEC validation status and interface scope for addresses as necessary for supporting link-local networking).
• The glibc getaddrinfo(3) API as defined by RFC3493 and its related resolver functions, including gethostbyname(3). This API is widely supported, including beyond the Linux platform. In its current form it does not expose DNSSEC validation status information however, and is synchronous only. This API is backed by the glibc Name Service Switch (nss(5)). Usage of the glibc NSS module nss-resolve(8) is required in order to allow glibc's NSS resolver functions to resolve host names via systemd-resolved.
• Additionally, systemd-resolved provides a local DNS stub listener on IP address 127.0.0.53 on the local loopback interface. Programs issuing DNS requests directly, bypassing any local API may be directed to this stub, in order to connect them to systemd-resolved. Note however that it is strongly recommended that local programs use the glibc NSS or bus APIs instead (as described above), as various network resolution concepts (such as link-local addressing, or LLMNR Unicode domains) cannot be mapped to the unicast DNS protocol.

[angystardust]

Since Fedora33 went GA, we now have systemd-resolved enabled by default so we need to find a proper way to manage this DNS mess in a sane way. Actually, one needs to manually call resolvconf to inject the search-dns and dns-address parameters so imho it's far from being a nice end-user experience. Maybe the work done in #615 can be a first step to improve this issue, isn't it?

[peterhoeg]

Actually, one needs to manually call resolvconf to inject the search-dns

I presume you meant to say resolvectl instead.

As in case somebody else needs this, the exact invocation is:

resolvectl dns ppp0 1.2.3.4 resolvectl domain ppp0 example.com

[angystardust]

@peterhoeg on fedora you can also inject them using resolvconf -a ppp0 < ~/you-own-personal-resolv.confso passing the parameters defined in a custom resolv.conf