dns-proxy-server icon indicating copy to clipboard operation
dns-proxy-server copied to clipboard
verified publisher icon mageddo

Bug: systemd-resolved issues

Open ratcashdev opened this issue 1 year ago • 7 comments

What is Happening

starting dns-proxy from within a compose file, like:

services:
  dns-proxy:
    image: defreitas/dns-proxy-server
    hostname: dns
    volumes:
      - type: bind
        source: /var/run/docker.sock
        target: /var/run/docker.sock
      - type: bind
        source: /etc
        target: /host/etc
    ports:
      - "5380:5380"

ping dns executed from the host says Name or service not known The logs, in addition contain the entry: Not possible to restart resolved service: service systemd-resolved restart

Actually, the command service is not present on my machine. Only systemctl restart systemd-resolved works. However, even if I restart resolved manually, DNS lookup does not work through dns-proxy.

What is Expected

systemd-resolved is restarted and DNS lookup works.

Specs

  • Docker Version: docker version 27.1.2
  • DPS Version:
    • When using docker: the docker image version or run the following if you are using the latest docker run --rm defreitas/dns-proxy-server --version 3.25.10-snapshot
  • OS: Linux mymachine 6.10.6-10-MANJARO 1 SMP PREEMPT_DYNAMIC Sat Aug 24 16:17:40 UTC 2024 x86_64 GNU/Linux
  • Attach DPS Log file
12:03:31.664 [main           ] DEB c.m.d.config.dataprovider.ConfigPathBuilder       l=37   m=build                           status=configPathBuilt, path=/app/conf/config.json
12:03:31.675 [main           ] INF c.m.dnsproxyserver.config.dataprovider.JsonConfigsl=69   m=createDefaultConfigOnPath       status=createdDefaultConfigFile, path=/app/conf/config.json
12:03:31.676 [main           ] DEB c.m.d.config.dataprovider.ConfigDAOJson           l=32   m=find                            configPath=/app/conf/config.json
12:03:31.698 [main           ] INF c.m.d.s.d.a.DpsDockerEnvironmentSetupService      l=32   m=setup                           status=binding-docker-events, connectedToDocker=true
12:03:31.698 [main           ] INF c.m.d.s.d.a.DpsDockerEnvironmentSetupService      l=44   m=setupNetwork                    status=dpsNetwork, active=false
12:03:31.698 [main           ] INF c.m.d.s.docker.application.DpsContainerService    l=116  m=tRunningContainersToDpsNetwork  status=autoConnectDpsNetworkDisabled, dpsNetwork=false, dpsNetworkAutoConnect=false
12:03:31.698 [main           ] INF c.m.d.solver.docker.entrypoint.EventListener      l=32   m=onStart                         status=containerAutoConnectToDpsNetworkDisabled
12:03:31.698 [main           ] INF com.mageddo.dnsserver.UDPServerPool               l=31   m=start                           Starting UDP server, addresses=/0.0.0.0:53
12:03:31.698 [main           ] INF c.mageddo.dnsproxyserver.server.dns.ServerStarter l=30   m=start                           status=startingDnsServer, protocol=UDP_TCP, port=53
12:03:31.701 [virtual-44     ] INF com.mageddo.dnsserver.TCPServer                   l=56   m=start0                          status=tcpServerStarting, port=53
12:03:31.706 [main           ] INF com.mageddo.http.WebServer                        l=134  m=start                           status=startingWebServer, port=5380
12:03:36.733 [Thread-3       ] INF c.m.d.dnsconfigurator.DnsConfigurators            l=113  m=getInstance0                    usingDnsConfigurator=DnsConfiguratorLinux
12:03:36.733 [Thread-3       ] INF c.m.d.dnsconfigurator.linux.DnsConfiguratorLinux  l=122  m=init                            status=using, configFile=ResolvFile(path=/host/etc/systemd/resolved.conf, type=SYSTEMD_RESOLVED)
12:03:36.738 [Thread-3       ] WAR c.m.d.dnsconfigurator.linux.DnsConfiguratorLinux  l=145  m=tryRestartResolved              status=can't restart resolved service, please run: 'service systemd-resolved restart' to apply DPS as default DNS.
Not possible to restart resolved service: 1 : systemd-resolved: unrecognized service

ratcashdev avatar Sep 30 '24 12:09 ratcashdev

Hey @ratcashdev , DPS won´t be able to configure itself as the default DNS when running inside a docker container when the host is using system-resolved.

Yep, you will do need to restart systemd-resolved using service or systemctl command, see more details about the limitation.

If you are using DPS as your primary DNS solver I would recommend follow theses steps to configure it as a service and don't have to reconfigure systemd-resolved at every restart.

In alternative you can bypass systemd-resolved and use resolv.conf without the need to restart any service by following these instructions:

services:
  dns-proxy:
    image: defreitas/dns-proxy-server
    hostname: dns
    volumes:
      - type: bind
        source: /var/run/docker.sock
        target: /var/run/docker.sock
      - type: bind
        source: /etc
        target: /host/etc
   environment: 
     - MG_RESOLVCONF=/host/etc/resolv.conf
    ports:
      - "5380:5380"

mageddo avatar Sep 30 '24 14:09 mageddo

HI @mageddo thanks for the quick replay and solution.

DPS won´t be able to configure itself as the default DNS when running inside a docker container when the host is using system-resolved.

can you plz elaborate what is the fundamental issue here?

ratcashdev avatar Sep 30 '24 14:09 ratcashdev

Basically systemd-resolved need to be restarted and it's a host program, a host program can't be executed from the docker container as expected by the isolation principle, DPS is running inside a docker container so it can´t restart the host system-resolved at the host.

mageddo avatar Sep 30 '24 15:09 mageddo

@mageddo I have tried your suggested setup above. Unfortunately, it seems it still does not work. here are the logs:

dns-proxy-1  | 06:41:24.864 [main           ] DEB c.m.d.config.dataprovider.ConfigPathBuilder       l=37   m=build                           status=configPathBuilt, path=/app/conf/config.json
dns-proxy-1  | 06:41:24.874 [main           ] INF c.m.dnsproxyserver.config.dataprovider.JsonConfigsl=69   m=createDefaultConfigOnPath       status=createdDefaultConfigFile, path=/app/conf/config.json
dns-proxy-1  | 06:41:24.874 [main           ] DEB c.m.d.config.dataprovider.ConfigDAOJson           l=32   m=find                            configPath=/app/conf/config.json
dns-proxy-1  | 06:41:24.890 [main           ] INF c.m.d.s.d.a.DpsDockerEnvironmentSetupService      l=32   m=setup                           status=binding-docker-events, connectedToDocker=true
dns-proxy-1  | 06:41:24.890 [main           ] INF c.m.d.s.d.a.DpsDockerEnvironmentSetupService      l=44   m=setupNetwork                    status=dpsNetwork, active=false
dns-proxy-1  | 06:41:24.890 [main           ] INF c.m.d.s.docker.application.DpsContainerService    l=116  m=tRunningContainersToDpsNetwork  status=autoConnectDpsNetworkDisabled, dpsNetwork=false, dpsNetworkAutoConnect=false
dns-proxy-1  | 06:41:24.890 [main           ] INF c.m.d.solver.docker.entrypoint.EventListener      l=32   m=onStart                         status=containerAutoConnectToDpsNetworkDisabled
dns-proxy-1  | 06:41:24.890 [main           ] INF com.mageddo.dnsserver.UDPServerPool               l=31   m=start                           Starting UDP server, addresses=/0.0.0.0:53
dns-proxy-1  | 06:41:24.890 [main           ] INF c.mageddo.dnsproxyserver.server.dns.ServerStarter l=30   m=start                           status=startingDnsServer, protocol=UDP_TCP, port=53
dns-proxy-1  | 06:41:24.892 [virtual-44     ] INF com.mageddo.dnsserver.TCPServer                   l=56   m=start0                          status=tcpServerStarting, port=53
dns-proxy-1  | 06:41:24.897 [main           ] INF com.mageddo.http.WebServer                        l=134  m=start                           status=startingWebServer, port=5380
dns-proxy-1  | 06:41:29.911 [Thread-2       ] INF c.m.d.dnsconfigurator.DnsConfigurators            l=113  m=getInstance0                    usingDnsConfigurator=DnsConfiguratorLinux
dns-proxy-1  | 06:41:29.911 [Thread-2       ] INF c.m.d.dnsconfigurator.linux.DnsConfiguratorLinux  l=91   m=lambda$findBestConfFile$0       status=noValidConfFile, file=/host/etc/resolv.conf
dns-proxy-1  | 06:41:29.911 [Thread-2       ] INF c.m.d.dnsconfigurator.linux.DnsConfiguratorLinux  l=122  m=init                            status=using, configFile=null

/etc/resolv.conf still stays the same, containing nameserver 127.0.0.53

ratcashdev avatar Oct 01 '24 06:10 ratcashdev

I will do a check, looks like DPS hasn't permission to edit resolv.conf

mageddo avatar Oct 01 '24 15:10 mageddo

yeah, pretty much:

-rw-r--r-- 1 systemd-resolve systemd-resolve 920 sep 30 16:23 /run/systemd/resolve/stub-resolv.conf

ratcashdev avatar Oct 01 '24 17:10 ratcashdev

@ratcashdev

Got it.

/host/etc/resolv.conf is a symlink

# ls -lha /host/etc/resolv*
lrwxrwxrwx 1 root root   39 Sep 15  2022 /host/etc/resolv.conf -> ../run/systemd/resolve/stub-resolv.conf

So you need to run like below:

services:
  dns-proxy:
    image: defreitas/dns-proxy-server
    hostname: dns
    volumes:
      - /run/systemd/resolve/:/host/run/
    environment:
      MG_RESOLVCONF: /host/run/stub-resolv.conf
    ports:
      - "5380:5380"


dns-proxy-1  | 13:00:31.189 [Thread-3       ] INF c.m.d.dnsconfigurator.linux.DnsConfiguratorLinux  l=122  m=init                            status=using, configFile=ResolvFile(path=/host/run/stub-resolv.conf, type=RESOLVCONF)

Improving the logs to make the failing condition explicit at #585 .

mageddo avatar Oct 07 '24 13:10 mageddo

This issue is stale because it has been waiting-feedback for 15 days with no activity.

github-actions[bot] avatar Oct 23 '24 07:10 github-actions[bot]

This issue was closed because it has been inactive for 7 days since being marked as stale, you can reopen it at any time.

github-actions[bot] avatar Oct 31 '24 07:10 github-actions[bot]

Tested this today. Unfortunately, it still does not work.

Logs look good, but a simple ping to a hostname registered in the compose file fails.

logs:

dns-proxy-1  | 09:20:45.529 [main           ] DEB c.m.d.config.dataprovider.ConfigPathBuilder       l=37   m=build                           status=configPathBuilt, path=/app/conf/config.json
dns-proxy-1  | 09:20:45.536 [main           ] INF c.m.dnsproxyserver.config.dataprovider.JsonConfigsl=69   m=createDefaultConfigOnPath       status=createdDefaultConfigFile, path=/app/conf/config.json
dns-proxy-1  | 09:20:45.537 [main           ] DEB c.m.d.config.dataprovider.ConfigDAOJson           l=32   m=find                            configPath=/app/conf/config.json
dns-proxy-1  | 09:20:45.553 [main           ] INF c.m.d.s.d.a.DpsDockerEnvironmentSetupService      l=32   m=setup                           status=binding-docker-events, connectedToDocker=true
dns-proxy-1  | 09:20:45.553 [main           ] INF c.m.d.s.d.a.DpsDockerEnvironmentSetupService      l=44   m=setupNetwork                    status=dpsNetwork, active=false
dns-proxy-1  | 09:20:45.553 [main           ] INF c.m.d.s.docker.application.DpsContainerService    l=116  m=tRunningContainersToDpsNetwork  status=autoConnectDpsNetworkDisabled, dpsNetwork=false, dpsNetworkAutoConnect=false
dns-proxy-1  | 09:20:45.553 [main           ] INF c.m.d.solver.docker.entrypoint.EventListener      l=32   m=onStart                         status=containerAutoConnectToDpsNetworkDisabled
dns-proxy-1  | 09:20:45.554 [main           ] INF com.mageddo.dnsserver.UDPServerPool               l=31   m=start                           Starting UDP server, addresses=/0.0.0.0:53
dns-proxy-1  | 09:20:45.554 [main           ] INF c.mageddo.dnsproxyserver.server.dns.ServerStarter l=30   m=start                           status=startingDnsServer, protocol=UDP_TCP, port=53
dns-proxy-1  | 09:20:45.556 [virtual-44     ] INF com.mageddo.dnsserver.TCPServer                   l=56   m=start0                          status=tcpServerStarting, port=53
dns-proxy-1  | 09:20:45.562 [main           ] INF com.mageddo.http.WebServer                        l=134  m=start                           status=startingWebServer, port=5380
dns-proxy-1  | 09:20:50.603 [Thread-3       ] INF c.m.d.dnsconfigurator.DnsConfigurators            l=113  m=getInstance0                    usingDnsConfigurator=DnsConfiguratorLinux
dns-proxy-1  | 09:20:50.604 [Thread-3       ] INF c.m.d.dnsconfigurator.linux.DnsConfiguratorLinux  l=122  m=init                            status=using, configFile=ResolvFile(path=/host/run/stub-resolv.conf, type=RESOLVCONF)

/etc/resolv.conf (linked to /run/systemd/resolve/stub-resolv.conf) now looks like this:

# nameserver 127.0.0.53 # dps-comment
options edns0 trust-ad
search .
nameserver 172.18.0.2 # dps-entry

ratcashdev avatar Oct 31 '24 09:10 ratcashdev

@ratcashdev can you confirm that DPS is being used as the default DNS? can you see query entries at DPS logs?

If your /etc/resolv.conf keeps with nameserver 172.18.0.2 # dps-entry then it's supposed to work.

You can ping dps-sample.dev or host.docker to check if DPS is working.

mageddo avatar Oct 31 '24 15:10 mageddo

@mageddo unfortunately none of the pings work and I don't see any name queries in the DPS logs.

ratcashdev avatar Oct 31 '24 16:10 ratcashdev

Looks like systemd-resolved is acting and ignoring resolv.conf. If you really want to run DPS inside docker container you can set an static IP for DPS container and configure it on systemd-resolved as following (you will need to use DPS 3.30.5-snapshot):

services:
  dns-proxy:
    image: defreitas/dns-proxy-server:3.30.5-snapshot
    hostname: dns
    volumes:
      - type: bind
        source: /var/run/docker.sock
        target: /var/run/docker.sock
    networks:
      my_network:
        ipv4_address: 171.18.0.10  
    environment:
      - MG_LOG_LEVEL=DEBUG

networks:
  my_network:
    driver: bridge
    ipam:
      config:
        - subnet: 171.18.0.0/16

sudo nano /etc/systemd/resolved.conf

Locate the DNS= line in the [Resolve] section and add the container's static IP:

[Resolve]
DNS=171.18.0.10 
FallbackDNS=

Restart the systemd-resolved service to apply the changes: DPS will be used when available

sudo systemctl restart systemd-resolved

mageddo avatar Oct 31 '24 18:10 mageddo

Thanks for the suggestion. My primary use case was to start dns-proxy from inside docker-compose to allow seamless communication between the host and other containers using host names during integration testing. The requirement to restart resolved makes this a non-practical approach (to the extent of being practically unusable) for my use case. Thanks for your efforts, either way!

ratcashdev avatar Nov 11 '24 11:11 ratcashdev