Hundreds of DNS queries for mariadb.lan hitting outside DNS server

[algotechio]

On my network's pihole, I found thousands of DNS queries hitting the pihole DNS server:

The queries were coming from umbrel server.

The issue was that mariadb container was not started on the umbrel server. Then the mariadb was not getting resolved by docker's DNS and instead forwarded the DNS resolution to the host.

I searched the code and it seems that the only place that uses docker host naming is this line (the other places seem to use direct IP addresses):

https://github.com/getumbrel/umbrel/blob/55d77495135ecfdd2060544ff3a7656822aba727/apps/mempool/docker-compose.yml#L25

[algotechio]

Maybe even adding DNS flag to each container that has no business connecting to the outside world. This should make it safer and even prevent some rogue upstream dependency connecting to the internet.

[nevets963]

@algotechio If you're sure that the requests are coming from your Umbrel server, then I assume you actually have the Mempool app installed? If so, it's certainly strange that Docker didn't resolve DNS here for the mariadb container. Best guess would be; maybe the mariadb container failed to boot for some reason? And the wait-for-it.sh script is endlessly checking for mariadb to come online. Are you able to confirm if you have Mempool installed and whether or not you see any problems with Mempool's mariadb instance?

We can solve this by using mariadb container IP directly, but I'm curious if there is another issue further upstream.

[algotechio]

Hello,

to recreate the problem:

1. So in the right now I have all the services running and there are no DNS queries hitting the outside pihole.

sudo docker container ls
...
8704d2cb64e7   mariadb:10.5.12            "docker-entrypoint.s…"    2 weeks ago   Up 3 days          3306/tcp         mempool_mariadb_1
...

2. Then I reboot the server (running inside VM ubuntu 20.04 server):

3. After reboot, the mariadb does not seem to be running

sudo docker container ls
...
8704d2cb64e7   mariadb:10.5.12                        "docker-entrypoint.s…"    2 weeks ago   Exited (0) 8 minutes ago                                                             mempool_mariadb_1
...

# and checking the logs confirms that the database is not running
sudo docker container logs mempool_mariadb_1 

2022-03-31  6:47:53 4 [Warning] Aborted connection 4 to db: 'unconnected' user: 'unauthenticated' host: '10.21.21.27' (This connection closed normally without authentication)
2022-03-31  6:47:54 0 [Note] InnoDB: Buffer pool(s) load completed at 220331  6:47:54
2022-04-04  0:39:46 5 [Warning] Aborted connection 5 to db: 'mempool' user: 'mempool' host: '10.21.21.27' (Got an error reading communication packets)
2022-04-04  0:39:47 0 [Note] mysqld (initiated by: unknown): Normal shutdown
2022-04-04  0:39:47 0 [Note] Event Scheduler: Purging the queue. 0 events
2022-04-04  0:39:47 0 [Note] InnoDB: FTS optimize thread exiting.
2022-04-04  0:39:47 0 [Note] InnoDB: Starting shutdown...
2022-04-04  0:39:47 0 [Note] InnoDB: Dumping buffer pool(s) to /var/lib/mysql/ib_buffer_pool
2022-04-04  0:39:47 0 [Note] InnoDB: Restricted to 2016 pages due to innodb_buf_pool_dump_pct=25
2022-04-04  0:39:48 0 [Note] InnoDB: Buffer pool(s) dump completed at 220404  0:39:48
2022-04-04  0:39:50 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
2022-04-04  0:39:50 0 [Note] InnoDB: Shutdown completed; log sequence number 168627177; transaction id 509096
2022-04-04  0:39:50 0 [Note] mysqld: Shutdown complete

4. Outside pihole is not hammered with lookups of mariadb.lan at a rate of ~2 per second

5. Running the start script fixes the problem:

sudo ./scripts/start

6. mariadb is now running and the DNS queries in outside pihole are gone.

I think the issue is that after the reboot the containers are not brought up:

1. ./umbrel/apps/mempool/docker-compose.yml restarts maribdb only on-failure (instead of unless-stopped):

  mariadb:
    image: mariadb:10.5.12@sha256:dfcba5641bdbfd7cbf5b07eeed707e6a3672f46823695a0d3aba2e49bbd9b1dd
    user: "1000:1000"
    restart: on-failure
    stop_grace_period: 1m
   ...

2. The ./umbrel/scripts/start does not seem to be run by systemctl on bootup.

[onepabz]

Can confirm the same behaviour in my umbrel node

[jeffreyjackson]

also noticed this behavior with my umbrel node, but it eventually stopped.