docker-pi-hole icon indicating copy to clipboard operation
docker-pi-hole copied to clipboard

Pi Hole containers cannot resolve DNS when using docker bridge network

Open thcipriani opened this issue 2 years ago • 16 comments

This is a: Bug

Details

When using a user-defined bridge network (which is the default with docker-compose) the pi hole container cannot resolve DNS (although it can serve DNS traffic from outside the bridge network).

This means Gravity update fails: Gravity update fails to resolve DNS

Related Issues

  • [x] I have searched this repository/Pi-hole forums for existing issues and pull requests that look similar
  • #945
  • #639
  • #497 Also, googling "127.0.0.11 pihole" comes back with a bunch of related issues:
  • https://discourse.pi-hole.net/t/127-0-0-11-in-etc-resolv-conf/22450
  • https://stackoverflow.com/a/69994721/2812207
  • https://unix.stackexchange.com/questions/647996/docker-container-dns-not-working-with-pihole
  • https://www.reddit.com/r/pihole/comments/v7wyqm/messed_up_adlist_and_gravity_will_not_update_how/

How to reproduce the issue

  1. Environment data
  • Operating System: Debian 11 Bullseye
  • Hardware: qemu VM running on a PC
  • Kernel Architecture: x86_64
  • Docker Install Info and version:
    • Software source: apt installed: 20.10.5+dfsg1-1+b5 in Debian bullseye
    • Supplimentary Software: docker-compose 1.25.0-1
  • Hardware architecture: x86
  1. docker-compose.yml contents, docker run shell command, or paste a screenshot of any UI based configuration of containers here

I run this with docker-compose up -d

version: "3"

# https://github.com/pi-hole/docker-pi-hole/blob/master/README.md

services:
  pihole:
    container_name: pihole
    image: pihole/pihole:latest
    ports:
      - "53:53/tcp"
      - "53:53/udp"
      - "67:67/udp" # Only required if you are using Pi-hole as your DHCP server
      - "80:80/tcp"
    environment:
      TZ: 'America/Denver'
    # Volumes store your data between container upgrades
    volumes:
      - './etc-pihole:/etc/pihole'
      - './etc-dnsmasq.d:/etc/dnsmasq.d'
      - './var-log/pihole.log:/var/log/pihole.log'
    # Recommended but not required (DHCP needs NET_ADMIN)
    #   https://github.com/pi-hole/docker-pi-hole#note-on-capabilities
    cap_add:
      - NET_ADMIN
    restart: unless-stopped
  1. any additional info to help reproduce

These common fixes didn't work for my issue

  • [x] I have tried removing/destroying my container, and re-creating a new container
  • [x] I have tried fresh volume data by backing up and moving/removing the old volume data
    • NA
  • [x] I have tried running the stock docker run example(s) in the readme (removing any customizations I added)

A simple docker run fails on the same host, likely also due to weird iptables mangling:

thcipriani@docker:~$ docker run --rm -it  --entrypoint /bin/bash pihole/pihole:latest
root@dcc26ac5f674:/# nslookup google.com
;; reply from unexpected source: 172.17.0.1#53, expected 172.16.1.56#53

;; reply from unexpected source: 172.17.0.1#53, expected 172.16.1.56#53

;; reply from unexpected source: 172.17.0.1#53, expected 172.16.1.56#53

;; connection timed out; no servers could be reached
root@dcc26ac5f674:/# echo 'nameserver 8.8.8.8' > /etc/resolv.conf 
root@dcc26ac5f674:/# nslookup google.com
Server:         8.8.8.8
Address:        8.8.8.8#53

Non-authoritative answer:
Name:   google.com
Address: 142.250.217.110
Name:   google.com
Address: 2607:f8b0:400a:80a::200e

None of the other containers on that host work either. If I run this same command on another host in my LAN: works fine. Even with the 127.0.0.11 nameserver.

  • [x] I have tried a newer or older version of Docker Pi-hole (depending what version the issue started in for me)
    • NA
  • [x] I have tried running without my volume data mounts to eliminate volumes as the cause
    • NA

If the above debugging / fixes revealed any new information note it here. Add any other debugging steps you've taken or theories on root cause that may help.

Wild speculation

When using a user-defined bridge network, the DNS nameserver inside containers is set to 127.0.0.11. Docker's built-in dns server resolves DNS and passes queries it cannot resolve to the host (i.e., if they're not the name of the other containers on the network).

This fails inside the pi-hole container. Other containers are also unable to resolve DNS. The host and the rest of the LAN are able to use pihole.

This may have something to do with docker's manipulation of iptables the nat table's DOCKER chain—I see its DNAT rules DNAT traffic NOT originating from the docker network's interface on port 53 to the container IP (which might mean everything but the DNS requests of the container). I tried fiddling with it to no avail.

Here's the docker bridge network 🌉

thcipriani@docker:~/pi-hole$ docker network inspect cf1535b51633
[
    {
        "Name": "pi-hole_default",
        "Id": "cf1535b516334720032bbcf0d49b20783c6107142b8f86dec2e87f0d4c360eb4",
        "Created": "2022-05-13T14:53:17.720023003-06:00",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "172.20.0.0/16",
                    "Gateway": "172.20.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": true,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "8816783b3218b9f82bbf5be522e6a61f056a988562cfb1cd4cbe36bb39a22521": {
                "Name": "pihole",
                "EndpointID": "5d86b4bb2e61e455ef43f06c3de770c4e3ad91f6dfdb4f818f35e030af88a1cf",
                "MacAddress": "02:42:ac:14:00:02",
                "IPv4Address": "172.20.0.2/16",
                "IPv6Address": ""
            }
        },
        "Options": {},
        "Labels": {
            "com.docker.compose.network": "default",
            "com.docker.compose.project": "pi-hole",
            "com.docker.compose.version": "1.25.0"
        }
    }
]

Here's the interface that bridge nework uses:

thcipriani@docker:~/pi-hole$ ip addr | grep -B 2 -A 3 172.20.0
4: br-cf1535b51633: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 02:42:f7:43:e2:de brd ff:ff:ff:ff:ff:ff
    inet 172.20.0.1/16 brd 172.20.255.255 scope global br-cf1535b51633
       valid_lft forever preferred_lft forever
    inet6 fe80::42:f7ff:fe43:e2de/64 scope link 
       valid_lft forever preferred_lft forever

And, finally, relevant iptables rules

thcipriani@squeeze:~/pi-hole$ sudo iptables-save | grep -P '(^:|br-cf1535b51633)'
:INPUT DROP [47:2820]
:FORWARD DROP [4:344]
:OUTPUT ACCEPT [15:644]
:DOCKER - [0:0]
:DOCKER-ISOLATION-STAGE-1 - [0:0]
:DOCKER-ISOLATION-STAGE-2 - [0:0]
:DOCKER-USER - [0:0]
:ufw-after-forward - [0:0]
:ufw-after-input - [0:0]
:ufw-after-logging-forward - [0:0]
:ufw-after-logging-input - [0:0]
:ufw-after-logging-output - [0:0]
:ufw-after-output - [0:0]
:ufw-before-forward - [0:0]
:ufw-before-input - [0:0]
:ufw-before-logging-forward - [0:0]
:ufw-before-logging-input - [0:0]
:ufw-before-logging-output - [0:0]
:ufw-before-output - [0:0]
:ufw-logging-allow - [0:0]
:ufw-logging-deny - [0:0]
:ufw-not-local - [0:0]
:ufw-reject-forward - [0:0]
:ufw-reject-input - [0:0]
:ufw-reject-output - [0:0]
:ufw-skip-to-policy-forward - [0:0]
:ufw-skip-to-policy-input - [0:0]
:ufw-skip-to-policy-output - [0:0]
:ufw-track-forward - [0:0]
:ufw-track-input - [0:0]
:ufw-track-output - [0:0]
:ufw-user-forward - [0:0]
:ufw-user-input - [0:0]
:ufw-user-limit - [0:0]
:ufw-user-limit-accept - [0:0]
:ufw-user-logging-forward - [0:0]
:ufw-user-logging-input - [0:0]
:ufw-user-logging-output - [0:0]
:ufw-user-output - [0:0]
-A FORWARD -o br-cf1535b51633 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o br-cf1535b51633 -j DOCKER
-A FORWARD -i br-cf1535b51633 ! -o br-cf1535b51633 -j ACCEPT
-A FORWARD -i br-cf1535b51633 -o br-cf1535b51633 -j ACCEPT
-A DOCKER -d 172.20.0.2/32 ! -i br-cf1535b51633 -o br-cf1535b51633 -p tcp -m tcp --dport 80 -j ACCEPT
-A DOCKER -d 172.20.0.2/32 ! -i br-cf1535b51633 -o br-cf1535b51633 -p udp -m udp --dport 67 -j ACCEPT
-A DOCKER -d 172.20.0.2/32 ! -i br-cf1535b51633 -o br-cf1535b51633 -p tcp -m tcp --dport 53 -j ACCEPT
-A DOCKER -d 172.20.0.2/32 ! -i br-cf1535b51633 -o br-cf1535b51633 -p udp -m udp --dport 53 -j ACCEPT
-A DOCKER-ISOLATION-STAGE-1 -i br-cf1535b51633 ! -o br-cf1535b51633 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-2 -o br-cf1535b51633 -j DROP
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:DOCKER - [0:0]
-A POSTROUTING -s 172.20.0.0/16 ! -o br-cf1535b51633 -j MASQUERADE
-A DOCKER -i br-cf1535b51633 -j RETURN
-A DOCKER ! -i br-cf1535b51633 -p tcp -m tcp --dport 80 -j DNAT --to-destination 172.20.0.2:80
-A DOCKER ! -i br-cf1535b51633 -p udp -m udp --dport 67 -j DNAT --to-destination 172.20.0.2:67
-A DOCKER ! -i br-cf1535b51633 -p tcp -m tcp --dport 53 -j DNAT --to-destination 172.20.0.2:53
-A DOCKER ! -i br-cf1535b51633 -p udp -m udp --dport 53 -j DNAT --to-destination 172.20.0.2:53

If I start a new container using the same network (docker run --rm -it --name asdf --network pi-hole_default debian) I can resolve the name asdf in the pihole container:

root@8816783b3218:/# nslookup google.com
;; connection timed out; no servers could be reached


root@8816783b3218:/# nslookup asdf
Server:         127.0.0.11
Address:        127.0.0.11#53

Non-authoritative answer:
Name:   asdf
Address: 172.20.0.3

root@8816783b3218:/# cat /etc/resolv.conf
nameserver 127.0.0.11
options ndots:0

thcipriani avatar Aug 12 '22 21:08 thcipriani

What do you have the Interface Listening Behavior set to on the web interface? Settings > DNS > Interface settings

dschaper avatar Aug 12 '22 22:08 dschaper

docker run --rm -it --entrypoint /bin/bash pihole/pihole:latest

I don't think using bash as entrypoint will work for testing. Pi-hole won't start.

rdwebdesign avatar Aug 12 '22 22:08 rdwebdesign

What do you have the Interface Listening Behavior set to on the web interface? Settings > DNS > Interface settings

I currently have this set to, "Respond only on interface eth0"

docker run --rm -it --entrypoint /bin/bash pihole/pihole:latest

I don't think using bash as entrypoint will work for testing. Pi-hole won't start.

I had pihole DNS running in a different container on the same host. My example was meant to show sibling containers are also affected. I was unclear what to try for the "docker run example(s) in the readme"

thcipriani avatar Aug 14 '22 21:08 thcipriani

attempted to recreate this from scratch today.

  1. Fresh VM, Debian 11, x86_64—did this twice to verify with a digital ocean droplet
  2. apt update && apt dist-upgrade && shutdown -r now
  3. apt install docker.io docker-compose
  4. Same docker-compose.yml
version: "3"

# More info at https://github.com/pi-hole/docker-pi-hole/ and https://docs.pi-hole.net/
services:
  pihole:
    container_name: pihole
    image: pihole/pihole:latest
    # For DHCP it is recommended to remove these ports and instead add: network_mode: "host"
    ports:
      - "53:53/tcp"
      - "53:53/udp"
      - "67:67/udp" # Only required if you are using Pi-hole as your DHCP server
      - "80:80/tcp"
    environment:
      TZ: 'America/Chicago'
      WEBPASSWORD: 'pihole issue 1166 ftw'
    # Volumes store your data between container upgrades
    volumes:
      - './etc-pihole:/etc/pihole'
      - './etc-dnsmasq.d:/etc/dnsmasq.d'
    #   https://github.com/pi-hole/docker-pi-hole#note-on-capabilities
    cap_add:
      - NET_ADMIN # Required if you are using Pi-hole as your DHCP server, else not needed
    restart: unless-stopped

  1. docker compose up -d (everything works: host dns, container dns, using host ip as dns)
  2. Set host /etc/resolv.conf to the host ip (everything works: host dns, container dns, using host ip as dns)
  3. docker-compose down
  4. docker-compose up -d

Gravity update and dns worked fine at step 5, but when I took the container down, and brought it back up: It failed.

DNS on the host works. Using the host as a DNS server from another host works. DNS within the container fails.

thcipriani avatar Aug 14 '22 23:08 thcipriani

~~Are you changing /etc/resolv.conf inside the container?~~

Ah, sorry missed the 'host' part. I'm not sure why you are changing that file if everything is working on step 5?

You might need to change the Interface Listening Behavior to a more open configuration. Try the other steps.

dschaper avatar Aug 15 '22 00:08 dschaper

~Are you changing /etc/resolv.conf inside the container?~

Ah, sorry missed the 'host' part. I'm not sure why you are changing that file if everything is working on step 5?

At home, I want pihole to serve DNS to the entire network. The router uses the pihole as the primary DNS, so the host uses it as the primary DNS: it's a bit of a chicken-and-egg situation.

In testing, I was recreating the problem of my home network.

You might need to change the Interface Listening Behavior to a more open configuration. Try the other steps.

I'll try a new test changing the interface listening behavior and report back.

Which other steps are you referring to?

Thanks for your time and attention!

thcipriani avatar Aug 15 '22 14:08 thcipriani

So. I just spent way too much time on this.

But I still don't know how to fix it.


First thing, step 5 in my recreation procedure is a red-herring.

The container DNS works at that point because dockerd doesn't seem to notice updates to /etc/resolv.conf.


Pihole does resolve DNS

Next, pihole DOES resolve DNS BUT the container does not accept the response.

Here's what tcpdump running within the container network namespace shows while I'm running dig google.com:

root@pihole-1116:~# ip netns exec d29511050724 tcpdump -v
tcpdump: listening on eth0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
^C23:21:14.295227 IP (tos 0x0, ttl 64, id 30808, offset 0, flags [DF], proto UDP (17), length 79)
    172.20.0.3.34616 > <host>.domain: 13241+ [1au] A? google.com. (51)
23:21:14.296468 IP (tos 0x0, ttl 64, id 21176, offset 0, flags [DF], proto UDP (17), length 83)
    172.20.0.1.domain > 172.20.0.3.34616: 13241 1/0/1 google.com. A 142.250.191.78 (55)
23:21:14.296487 IP (tos 0xc0, ttl 64, id 29017, offset 0, flags [none], proto ICMP (1), length 111)
    172.20.0.3 > 172.20.0.1: ICMP 172.20.0.3 udp port 34616 unreachable, length 91
        IP (tos 0x0, ttl 64, id 21176, offset 0, flags [DF], proto UDP (17), length 83)
    172.20.0.1.domain > 172.20.0.3.34616: 13241 1/0/1 google.com. A 142.250.191.78 (55)

So you can see the container 172.20.0.3 make a request to the DNS of the host on port 53 (172.20.0.3.34616 > <host>.domain). Then it gets a response from the docker bridge network complete with the IP of google.com(!). Then for some reason the container sends an ICMP packet to the bridge.

But it doesn't make it back to the container

I can watch this route through netfilter logging:

  1. Host receives request
Sep  6 22:43:44 pihole-1116 kernel: [ 7304.778352] **PREROUTING nat**IN=br-c825b6b35050 OUT= PHYSIN=veth9be4d06 MAC=02:42:e8:e4:35:4c:02:42:ac:14:00:03:08:00 SRC=172.20.0.3 DST=<host ip> LEN=79 TOS=0x00 PREC=0x00 TTL=64 ID=42343 DF PROTO=UDP SPT=33684 DPT=53 LEN=59
Sep  6 22:43:44 pihole-1116 kernel: [ 7304.783221] **DOCKER nat**IN=br-c825b6b35050 OUT= PHYSIN=veth9be4d06 MAC=02:42:e8:e4:35:4c:02:42:ac:14:00:03:08:00 SRC=172.20.0.3 DST=<host ip> LEN=79 TOS=0x00 PREC=0x00 TTL=64 ID=42343 DF PROTO=UDP SPT=33684 DPT=53 LEN=59
Sep  6 22:43:44 pihole-1116 kernel: [ 7304.788058] **INPUT**IN=br-c825b6b35050 OUT= PHYSIN=veth9be4d06 MAC=02:42:e8:e4:35:4c:02:42:ac:14:00:03:08:00 SRC=172.20.0.3 DST=<host ip> LEN=79 TOS=0x00 PREC=0x00 TTL=64 ID=42343 DF PROTO=UDP SPT=33684 DPT=53 LEN=59
  1. Bridge sends a request to pihole:
Sep  6 22:43:44 pihole-1116 kernel: [ 7304.798015] **OUTPUT nat**IN= OUT=br-c825b6b35050 SRC=172.20.0.1 DST=172.20.0.2 LEN=79 TOS=0x00 PREC=0x00 TTL=64 ID=35346 DF PROTO=UDP SPT=54200 DPT=53 LEN=59
Sep  6 22:43:44 pihole-1116 kernel: [ 7304.802123] **OUTPUT**IN= OUT=br-c825b6b35050 SRC=172.20.0.1 DST=172.20.0.2 LEN=79 TOS=0x00 PREC=0x00 TTL=64 ID=35346 DF PROTO=UDP SPT=54200 DPT=53 LEN=59
Sep  6 22:43:44 pihole-1116 kernel: [ 7304.806154] **POSTROUTING nat**IN= OUT=br-c825b6b35050 SRC=172.20.0.1 DST=172.20.0.2 LEN=79 TOS=0x00 PREC=0x00 TTL=64 ID=35346 DF PROTO=UDP SPT=54200 DPT=53 LEN=59
  1. Pihole responds to the bridge
Sep  6 22:43:44 pihole-1116 kernel: [ 7304.820358] **INPUT**IN=br-c825b6b35050 OUT= PHYSIN=vethad1d9ca MAC=02:42:e8:e4:35:4c:02:42:ac:14:00:02:08:00 SRC=172.20.0.2 DST=172.20.0.1 LEN=83 TOS=0x00 PREC=0x00 TTL=64 ID=36282 DF PROTO=UDP SPT=53 DPT=54200 LEN=63
  1. Then the bridge tries to send the response to the container
Sep  6 22:43:44 pihole-1116 kernel: [ 7304.840673] **OUTPUT nat**IN= OUT=br-c825b6b35050 SRC=172.20.0.1 DST=172.20.0.3 LEN=83 TOS=0x00 PREC=0x00 TTL=64 ID=52608 DF PROTO=UDP SPT=53 DPT=33684 LEN=63
Sep  6 22:43:44 pihole-1116 kernel: [ 7304.844913] **OUTPUT**IN= OUT=br-c825b6b35050 SRC=172.20.0.1 DST=172.20.0.3 LEN=83 TOS=0x00 PREC=0x00 TTL=64 ID=52608 DF PROTO=UDP SPT=53 DPT=33684 LEN=63
Sep  6 22:43:44 pihole-1116 kernel: [ 7304.849072] **POSTROUTING nat**IN= OUT=br-c825b6b35050 SRC=172.20.0.1 DST=172.20.0.3 LEN=83 TOS=0x00 PREC=0x00 TTL=64 ID=52608 DF PROTO=UDP SPT=53 DPT=33684 LEN=63
  1. The unexplained ICMP packet from the container to the bridge
Sep  6 22:43:44 pihole-1116 kernel: [ 7304.862592] **INPUT**IN=br-c825b6b35050 OUT= PHYSIN=veth9be4d06 MAC=02:42:e8:e4:35:4c:02:42:ac:14:00:03:08:00 SRC=172.20.0.3 DST=172.20.0.1 LEN=111 TOS=0x00 PREC=0xC0 TTL=64 ID=4091 PROTO=ICMP TYPE=3 CODE=3 [SRC=172.20.0.1 DST=172.20.0.3 LEN=83 TOS=0x00 PREC=0x00 TTL=64 ID=52608 DF PROTO=UDP SPT=53 DPT=33684 LEN=63 ]

I could set DNS on the host to be the local IP of the pihole container. I think that might work.

But this setup should work. I should be able to set the DNS of a machine to its public IP, have dnsmasq/pihole run in a container and expose the ports on the host, and have that container able to resolve DNS. It seems like it should be possible.

Is this missing a masquerade rule somewhere? Is that the problem?

thcipriani avatar Sep 06 '22 23:09 thcipriani

Here are the masquerade rules installed by docker, FWIW:

-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A POSTROUTING -s 172.20.0.0/16 ! -o br-c825b6b35050 -j MASQUERADE
-A POSTROUTING -s 172.20.0.2/32 -d 172.20.0.2/32 -p tcp -m tcp --dport 80 -j MASQUERADE
-A POSTROUTING -s 172.20.0.2/32 -d 172.20.0.2/32 -p udp -m udp --dport 67 -j MASQUERADE
-A POSTROUTING -s 172.20.0.2/32 -d 172.20.0.2/32 -p tcp -m tcp --dport 53 -j MASQUERADE
-A POSTROUTING -s 172.20.0.2/32 -d 172.20.0.2/32 -p udp -m udp --dport 53 -j MASQUERADE

iptables -t nat -I POSTROUTING -s 172.20.0.1/32 -d 172.20.0.3/32 -p udp -m udp --sport 53 -j MASQUERADE didn't seem to have an effect 😭

thcipriani avatar Sep 07 '22 00:09 thcipriani

Since you are using a user-defined network, why don't you try to use a macvlan network?

rdwebdesign avatar Sep 07 '22 04:09 rdwebdesign

I am able to resolve this but entering the pi-hole container with: docker exec -it [container ID] /bin/bash

Then I modify etc/resolv.conf

The default resolv.conf contains this:

nameserver 127.0.0.11 options ndots:0

I change it to this:

search home nameserver 127.0.0.1 options ndots:0

After making that change I am able to update gravity. Since this is done inside the container the fix has to be applied every time the container restarts.

rworkm avatar Sep 07 '22 13:09 rworkm

@thcipriani

I was reading this issue again and I think the problem is you are trying to use ports already in use, causing conflicts. Using different ports in your original compose file will probably solve your issues.

Can you please try this?

version: "3"

# https://github.com/pi-hole/docker-pi-hole/blob/master/README.md

services:
  pihole:
    container_name: pihole
    image: pihole/pihole:latest
    ports:
      - "5053:53/tcp"
      - "5053:53/udp"
      - "67:67/udp" # Only required if you are using Pi-hole as your DHCP server
      - "8080:80/tcp"
    environment:
      TZ: 'America/Denver'
    # Volumes store your data between container upgrades
    volumes:
      - './etc-pihole:/etc/pihole'
      - './etc-dnsmasq.d:/etc/dnsmasq.d'
      - './var-log/pihole.log:/var/log/pihole.log'
    # Recommended but not required (DHCP needs NET_ADMIN)
    #   https://github.com/pi-hole/docker-pi-hole#note-on-capabilities
    cap_add:
      - NET_ADMIN
    restart: unless-stopped

rdwebdesign avatar Sep 09 '22 21:09 rdwebdesign

Hi, I think I have the same issue here every time I restart the container (e.g. after update) or the whole system pi-hole states, it can't resolve any DNS. I can fix it manually by restating the DNS resolver via the settings menu.

Sorry I can't provide any logs currently at least not much more then above. But I can say I have quite a similar setting, using docker compose, not using piholes dhcp.

Mathes

mathesst avatar Sep 14 '22 11:09 mathesst

Hitting this issue as well. Pihole's deployed using docker-compose hence gets its own user-defined bridge network. The host DNS points to a floating IP that points back to the host (keepalived). Any other docker container deployed on the host with docker-compose can't resolve DNS properly (they also get their own respective user-defined bridge network). Looking at pihole query logs I can see the DNS queries arriving and being answered but they don't seem to reach back. Anything running on other machines resolve just fine. Not sure what more info/debugging I can produce than that of @thcipriani .

artivis avatar Sep 18 '22 22:09 artivis

I was just hitting this issue, exact same symptoms and configuration. I added dns entries to the docker-compose and now all is working.

This happened to be the only container I am running at the moment, so not sure if effects others.

here is my compose file

version: "3"

# More info at https://github.com/pi-hole/docker-pi-hole/ and https://docs.pi-hole.net/
services:
  pihole:
    container_name: pihole
    image: pihole/pihole:latest
    # For DHCP it is recommended to remove these ports and instead add: network_mode: "host"
    ports:
      - "53:53/tcp"
      - "53:53/udp"
      - "67:67/udp" # Only required if you are using Pi-hole as your DHCP server
      - "8080:8080/tcp"
    environment:
      TZ: 'America/Los_Angeles'
      WEBPASSWORD: '*******'
      IPv6: 'false'
      WEB_PORT: '8080'
    dns:
      - 1.1.1.1
      - 1.0.0.1
      - 8.8.8.8
      - 8.8.4.4
    # Volumes store your data between container upgrades
    volumes:
      - './etc-pihole:/etc/pihole'
      - './etc-dnsmasq.d:/etc/dnsmasq.d'
      #   https://github.com/pi-hole/docker-pi-hole#note-on-capabilities
    cap_add:
      - NET_ADMIN # Recommended but not required (DHCP needs NET_ADMIN)
    restart: unless-stopped

edit from another busybox container I am unable to resolve any network names.

edit 2 new version that solves my problem, binding port 53 to the IP address of the host and using the default bridge.

The problem seems to be the user defined bridge created, maybe?

version: "3"

# More info at https://github.com/pi-hole/docker-pi-hole/ and https://docs.pi-hole.net/
services:
  pihole:
    container_name: pihole
    image: pihole/pihole:latest
    network_mode: bridge
    # For DHCP it is recommended to remove these ports and instead add: network_mode: "host"
    ports:
      - "xx.xx.xx.xx:53:53/tcp"
      - "xx.xx.xx.xx:53:53/udp"
      - "67:67/udp" # Only required if you are using Pi-hole as your DHCP server
      - "8080:8080/tcp"
    environment:
      TZ: 'America/Los_Angeles'
      WEBPASSWORD: '*******'
      IPv6: 'false'
      WEB_PORT: '8080'
    # Volumes store your data between container upgrades
    volumes:
      - './etc-pihole:/etc/pihole'
      - './etc-dnsmasq.d:/etc/dnsmasq.d'
      #   https://github.com/pi-hole/docker-pi-hole#note-on-capabilities
    cap_add:
      - NET_ADMIN # Recommended but not required (DHCP needs NET_ADMIN)
    restart: unless-stopped

jbreiding avatar Oct 03 '22 05:10 jbreiding

This issue has been mentioned on Pi-hole Userspace. There might be relevant details there:

https://discourse.pi-hole.net/t/pihole-docker-compose-not-blocking-anything-but-queries-are-hitting-it/58415/11

pralor-bot avatar Oct 10 '22 14:10 pralor-bot

I believe my issue was of a similar nature. I had to add the dns section to my docker-compose:

    volumes:
      - './etc-pihole:/etc/pihole'
      - './etc-dnsmasq.d:/etc/dnsmasq.d'
    # these are supposedly needed, otherwise you will get 127.0.0.1 problems https://github.com/pi-hole/docker-pi-hole/issues/410#issuecomment-460515706 - still doesn't work
    dns:
      - 127.0.0.1
      - 1.1.1.1
    #   https://github.com/pi-hole/docker-pi-hole#note-on-capabilities
    restart: unless-stopped

I spent several hours trying to figure this one out. It would be very helpful if something like this was posted to the docs or part of the example compose file.

olinic avatar Oct 19 '22 03:10 olinic

https://discourse.pi-hole.net/t/solve-dns-resolution-in-other-containers-when-using-docker-pihole/31413

d-rez avatar Nov 11 '22 12:11 d-rez

This issue is stale because it has been open 30 days with no activity. Please comment or update this issue or it will be closed in 5 days.

github-actions[bot] avatar Dec 12 '22 08:12 github-actions[bot]

Commenting something to un-stale this

d-rez avatar Dec 12 '22 08:12 d-rez

Does this need un-staling? The OP has long since stopped replying (even to a tagged question...) And other people are suggesting the solution is to set the DNS of the container with the configuration flag. (Or set up a macvlan network as suggested in another comment by @rdwebdesign)

This certainly appears to be a docker configuration issue rather than a bug. Setting DNS of your container to is event a documented (optional) step on the readme:

image

What more needs doing here?

PromoFaux avatar Dec 12 '22 08:12 PromoFaux

The original issue was probably a config problem with ports (never confirmed or denied by the OP). Also, some links mentioned on the initial post are config problems.

My suggestion is to close this issue and ask users to open new issues, case by case.

rdwebdesign avatar Dec 12 '22 17:12 rdwebdesign

If anything this is something that needs to be (imo) addressed in docs clearly. This isn't really such a limited use case, I came across this a couple months ago.

The issue here is that with a container running without host network access, using a docker stack (i.e. combined with unbound for DNS over TLS) the container network automatically gets created

Adding 127.0.0.1 to pihole container docker dns config does not resolve the underlying issue here because this isn't what's effectively causing it. The issue is that traffic from other containers arrives from an unexpected (for DNS server) interface as the solution is outlined at https://discourse.pi-hole.net/t/solve-dns-resolution-in-other-containers-when-using-docker-pihole/31413

To explain let's set some assumptions:

  • your docker host has an IP of 10.0.3.14/24
  • pihole runs in a stack on port 53 within the container bound to all interfaces (unless configured differently), let's say on 172.19.0.2, docker gw being 172.19.0.1
  • docker runs its own dns server on 127.0.0.11
  • your router on your network advertises that the DNS server is at 10.0.3.14

127.0.0.11:53 => 10.0.3.14:53 => 172.19.0.1:53 => 172.19.0.2:53 => result is reply from unexpected source because we were expecting the server to be on 10.0.3.14 but 172.19.0.2 replied after docker was done with its NAT rules.

Now there are 2 scenarios:

  • DNS request is made within pihole to the default DNS server. This scenario can be worked around by setting dns to 127.0.0.1 since this avoids any docker iptables shenanigans
  • DNS request is made from another container on the same docker host. This scenario can't be worked around by setting dns config option. The solution here is as explained in the link above to make sure the container forwarded port is bound to a host IP / specific interface

Sorry if I'm not explaining this better, bit convoluted and I'm most likely not fully understanding the routing here, but hopefully this makes sense

d-rez avatar Dec 13 '22 02:12 d-rez

127.0.0.11:53 => 10.0.3.14:53 => 172.19.0.1:53 => 172.19.0.2:53 => result is reply from unexpected source because we were expecting the server to be on 10.0.3.14 but 172.19.0.2 replied after docker was done with its NAT rules.

Each docker network driver has its upsides and downsides.

Maybe all this can be avoided if the user chooses to use macvlan network. Each container will receive its own IP on the local network. The router will advertise Pi-hole container IP. No port forwarding needed. No NAT.

rdwebdesign avatar Dec 13 '22 03:12 rdwebdesign

This issue is stale because it has been open 30 days with no activity. Please comment or update this issue or it will be closed in 5 days.

github-actions[bot] avatar Jan 12 '23 08:01 github-actions[bot]

If anything this is something that needs to be (imo) addressed in docs clearly. This isn't really such a limited use case, I came across this a couple months ago.

The issue here is that with a container running without host network access, using a docker stack (i.e. combined with unbound for DNS over TLS) the container network automatically gets created

Adding 127.0.0.1 to pihole container docker dns config does not resolve the underlying issue here because this isn't what's effectively causing it. The issue is that traffic from other containers arrives from an unexpected (for DNS server) interface as the solution is outlined at https://discourse.pi-hole.net/t/solve-dns-resolution-in-other-containers-when-using-docker-pihole/31413

To explain let's set some assumptions:

  • your docker host has an IP of 10.0.3.14/24
  • pihole runs in a stack on port 53 within the container bound to all interfaces (unless configured differently), let's say on 172.19.0.2, docker gw being 172.19.0.1
  • docker runs its own dns server on 127.0.0.11
  • your router on your network advertises that the DNS server is at 10.0.3.14

127.0.0.11:53 => 10.0.3.14:53 => 172.19.0.1:53 => 172.19.0.2:53 => result is reply from unexpected source because we were expecting the server to be on 10.0.3.14 but 172.19.0.2 replied after docker was done with its NAT rules.

Now there are 2 scenarios:

  • DNS request is made within pihole to the default DNS server. This scenario can be worked around by setting dns to 127.0.0.1 since this avoids any docker iptables shenanigans
  • DNS request is made from another container on the same docker host. This scenario can't be worked around by setting dns config option. The solution here is as explained in the link above to make sure the container forwarded port is bound to a host IP / specific interface

Sorry if I'm not explaining this better, bit convoluted and I'm most likely not fully understanding the routing here, but hopefully this makes sense

This ^ is exactly right—this is a NAT problem.

And it even has the solution I used: "make sure the container forwarded port is bound to a host IP".

That is, I changed:

# Problem docker-compose.yml
services:
  pihole:
    container_name: pihole
    image: pihole/pihole:latest
    ports:
      - "53:53/tcp"
      - "53:53/udp"

To (in my setup—172.16.1.56 is public LAN ip of my pihole—this would be different for others)

# Work docker-compose.yml
services:
  pihole:
    container_name: pihole
    image: pihole/pihole:latest
    ports:
      - "172.16.1.56:53:53/tcp"
      - "172.16.1.56:53:53/udp"

Things I did NOT do:

  • change 127.0.0.11 inside the container—this would remove docker's container resolution entirely
  • use a macvlan network—seems like this may fix this problem as well, but I did not try it

Sorry for disappearing on my own issue—I wanted to do more testing and propose some kind of fix or documentation update here. But that kept slipping down the todo list.

I hope this reply is useful, at least.

thcipriani avatar Mar 03 '23 20:03 thcipriani

I was able to get this up and running successfully in docker on a Synology today. Thanks for clarifying this. It appears to be working great for me.

For others attempting it, you'll most likely need to map port 80 to a different port, and when accessing the UI, add /admin or else you'll get a 403 error. I didn't dig much more into the cause of that, and it is fine for me since I just bookmark it locally.

version: "3"

# https://github.com/pi-hole/docker-pi-hole/blob/master/README.md

services:
  pihole:
    container_name: pihole
    image: pihole/pihole:latest
    # For DHCP it is recommended to remove these ports and instead add: network_mode: "host"
    ports:
      - "YOUR_NAS_IP:53:53/tcp"
      - "YOUR_NAS_IP:53:53/udp"
      - "67:67/udp"
      - "YOUR_NAS_IP:8080:80/tcp"
    environment:
      TZ: 'America/Chicago'
      # WEBPASSWORD: 'set a secure password here or it will be random'
    # Volumes store your data between container upgrades
    volumes:
      - './data/etc-pihole:/etc/pihole'  # you must create the data/etc-pihole directory in Shared folders
      - './data/etc-dnsmasq.d:/etc/dnsmasq.d' # you must create the data/etc-dnsmasq.d directory in Shared folders
    #   https://github.com/pi-hole/docker-pi-hole#note-on-capabilities
    cap_add:
      - NET_ADMIN
    restart: unless-stopped # Recommended but not required (DHCP needs NET_ADMIN)

I'm gonna run this with one or two devices for a few days before pointing the whole-home router's DNS to it. Thanks again!

brock avatar Mar 19 '23 19:03 brock

I had a similar issue. My solution was to simply add "search host" to the /etc/resolv.conf of my Raspberry Pi. As such, the current version of /etc/resolv.conf (the version that fixed the problem) looks like this:

# Generated by NetworkManager
search host
search local
nameserver 192.168.1.99

davinci-tech avatar Jul 04 '24 03:07 davinci-tech