exec /s6-init: no such file or directory since 2022.09.4

[andrasg]

Tried to upgrade a pi-hole docker installation that has been running issue free for ages to latest version, but the container does not start. Started investigating and found out that all is good up to version 2022.09.3, but 2022.09.4 fails with exec /s6-init: no such file or directory.

This is a: Run issue

Details

I simplified my test to the following and it still fails.

andrasg@ubuntu-sz:~/volumemaps/pi-hole$ docker run --rm --dns=127.0.0.1 --dns=1.1.1.1 -e PIHOLE_DNS_=192.168.1.253 pihole/pihole:2022.09.4
WARNING: Localhost DNS setting (--dns=127.0.0.1) may fail in containers.
exec /s6-init: no such file or directory

Related Issues

• [X] I have searched this repository/Pi-hole forums for existing issues and pull requests that look similar

How to reproduce the issue

1. Environment data

• Operating System: Ubuntu 20.04.5 LTS (GNU/Linux 5.4.0-126-generic x86_64)
• Hardware: Proxmox VM (Proxmox version 7.2-11)
• Kernel Architecture: x86_64
• Docker Install Info and version:
  • Software source: official docker
  • Supplimentary Software: portainer is used
• Hardware architecture: x64

2. docker-compose.yml contents, docker run shell command, or paste a screenshot of any UI based configuration of containers here

My regular run command would be this. Up to, including version 2022.09.03 works fine

docker run -d \
    --name pihole \
    -p 192.168.1.21:53:53/tcp \
    -p 192.168.1.21:53:53/udp \
    -p 192.168.1.21:80:80 \
    -p 192.168.1.21:443:443 \
    -e TZ="Europe/Budapest" \
    -e PIHOLE_DNS_=192.168.1.253 \
    -v "/home/andrasg/volumemaps/pi-hole/etc-pihole/:/etc/pihole/" \
    -v "/home/andrasg/volumemaps/pi-hole/etc-dnsmasq.d/:/etc/dnsmasq.d/" \
    --dns=1.1.1.1 \
    --restart=unless-stopped \
    pihole/pihole:latest

But I can reproduce it with this too:

docker run --rm --dns=127.0.0.1 --dns=1.1.1.1 -e PIHOLE_DNS_=192.168.1.253 -p 192.168.1.21:53:53/tcp -p 192.168.1.21:53:53/udp -e FTLCONF_LOCAL_IPV4="127.0.0.1" pihole/pihole:2022.09.4

Or even:

docker run --rm --dns=127.0.0.1 --dns=1.1.1.1 pihole/pihole:2022.09.4

4. any additional info to help reproduce

These common fixes didn't work for my issue

• [X] I have tried removing/destroying my container, and re-creating a new container
• [X] I have tried fresh volume data by backing up and moving/removing the old volume data
• [X] I have tried running the stock docker run example(s) in the readme (removing any customizations I added)
• [X] I have tried a newer or older version of Docker Pi-hole (depending what version the issue started in for me)
• [X] I have tried running without my volume data mounts to eliminate volumes as the cause

If the above debugging / fixes revealed any new information note it here. Add any other debugging steps you've taken or theories on root cause that may help.

[PromoFaux]

Hmm, unable to repro on a debian 11 VM:

Click for Container startup log

adam@uk-ple-aw:~$ docker run --rm --dns=127.0.0.1 --dns=1.1.1.1 pihole/pihole:2022.09.4
WARNING: Localhost DNS setting (--dns=127.0.0.1) may fail in containers.
Unable to find image 'pihole/pihole:2022.09.4' locally
2022.09.4: Pulling from pihole/pihole
31b3f1ad4ce1: Pull complete
2cf454ba7aa0: Pull complete
4f4fb700ef54: Pull complete
d5d1e33c70cc: Pull complete
82b13da8209d: Pull complete
48e71cb85034: Pull complete
Digest: sha256:aa6140856dfc67f3ae36c352e30903f944e4e220699ffeeaf6b85235b2d84c95
Status: Downloaded newer image for pihole/pihole:2022.09.4
s6-rc: info: service s6rc-oneshot-runner: starting
s6-rc: info: service s6rc-oneshot-runner successfully started
s6-rc: info: service fix-attrs: starting
s6-rc: info: service fix-attrs successfully started
s6-rc: info: service legacy-cont-init: starting
s6-rc: info: service legacy-cont-init successfully started
s6-rc: info: service cron: starting
s6-rc: info: service cron successfully started
s6-rc: info: service _uid-gid-changer: starting
s6-rc: info: service _uid-gid-changer successfully started
s6-rc: info: service _startup: starting
  [i] Starting docker specific checks & setup for docker pihole/pihole
  [i] Setting capabilities on pihole-FTL where possible
  [i] Applying the following caps to pihole-FTL:
        * CAP_CHOWN
        * CAP_NET_BIND_SERVICE
        * CAP_NET_RAW
  [i] Ensuring basic configuration by re-running select functions from basic-install.sh

  [i] Installing configs from /etc/.pihole...
  [i] Existing dnsmasq.conf found... it is not a Pi-hole file, leaving alone!
  [✓] Installed /etc/dnsmasq.d/01-pihole.conf
  [✓] Installed /etc/dnsmasq.d/06-rfc6761.conf

  [i] Installing latest logrotate script...
        [i] Existing logrotate file found. No changes made.
  [i] Assigning random password: lpPHxbbK
  [✓] New password set
  [i] Added ENV to php:
                        "TZ" => "",
                        "PIHOLE_DOCKER_TAG" => "2022.09.4",
                        "PHP_ERROR_LOG" => "/var/log/lighttpd/error-pihole.log",
                        "CORS_HOSTS" => "",
                        "VIRTUAL_HOST" => "0.0.0.0",
  [i] Using IPv4 and IPv6
  [i] setup_blocklists now setting default blocklists up:
  [i] TIP: Use a docker volume for /etc/pihole/adlists.list if you want to customize for first boot
  [i] Blocklists (/etc/pihole/adlists.list) now set to:
https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
  [i] Existing DNS servers detected in setupVars.conf. Leaving them alone
  [i] Applying pihole-FTL.conf setting LOCAL_IPV4=0.0.0.0
  [i] FTL binding to default interface: eth0
  [i] Enabling Query Logging
  [i] Testing lighttpd config: Syntax OK
  [i] All config checks passed, cleared for startup ...
  [i] Docker start setup complete

  Pi-hole version is v5.12.2 (Latest: v5.13)
  AdminLTE version is v5.15.1 (Latest: v5.16)
  FTL version is v5.18.1 (Latest: v5.18.2)
  Container tag is: 2022.09.4

  [i] pihole-FTL (no-daemon) will be started as pihole

s6-rc: info: service _startup successfully started
s6-rc: info: service pihole-FTL: starting
s6-rc: info: service pihole-FTL successfully started
s6-rc: info: service lighttpd: starting
s6-rc: info: service lighttpd successfully started
s6-rc: info: service _gravityonboot: starting
s6-rc: info: service _gravityonboot successfully started
  Checking if custom gravity.db is set in /etc/pihole/pihole-FTL.conf
s6-rc: info: service legacy-services: starting
s6-rc: info: service legacy-services successfully started
  [i] Neutrino emissions detected...
  [✓] Pulling blocklist source list into range

  [✓] Preparing new gravity database
  [i] Using libz compression

  [i] Target: https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
  [✓] Status: Retrieval successful
  [i] Analyzed 152588 domains
  [i] List has been updated

  [✓] Creating new gravity databases
  [✓] Storing downloaded domains in new gravity database
  [✓] Building tree
  [✓] Swapping databases
  [✓] The old database remains available.
  [i] Number of gravity domains: 152588 (152588 unique domains)
  [i] Number of exact blacklisted domains: 0
  [i] Number of regex blacklist filters: 0
  [i] Number of exact whitelisted domains: 0
  [i] Number of regex whitelist filters: 0
  [✓] Cleaning up stray matter

  [✓] FTL is listening on port 53
     [✓] UDP (IPv4)
     [✓] TCP (IPv4)
     [✓] UDP (IPv6)
     [✓] TCP (IPv6)

  [i] Pi-hole blocking will be enabled
  [i] Enabling blocking
  [✓] Pi-hole Enabled

I don't currently have access to a Ubuntu vm - and don't have proxmox so unable to test these things at the moment.

Clutching at straws here, but can you try starting up without the --dns=1.1.1.1 option? I note there is a warning - but again, clutching at straws!

[andrasg]

Unfortunately the same. In the meantime, I have upgraded Ubuntu to 22.04.1 but no change. Interestingly 2022.09.3 works without an issue. Also cannot wrap my head around the no such file error.

Is there any way to launch pihole in verbose mode to see what s6-init might be doing when encountering this issue?

andrasg@ubuntu-sz:~$ docker run --rm --dns=127.0.0.1 -e PIHOLE_DNS_=192.168.1.253 pihole/pihole:2022.09.4
WARNING: Localhost DNS setting (--dns=127.0.0.1) may fail in containers.
exec /s6-init: no such file or directory

[PromoFaux]

You can try -e PH_VERBOSE=1 - lets see what happens I guess!

[andrasg]

andrasg@ubuntu-sz:~$ docker run --rm --dns=127.0.0.1 -e PIHOLE_DNS_=192.168.1.253 -e PH_VERBOSE=1 pihole/pihole:2022.09.4
WARNING: Localhost DNS setting (--dns=127.0.0.1) may fail in containers.
exec /s6-init: no such file or directory

No change. I feel I am missing something obvious here. Have another Ubuntu VM and there 2022.10 runs without an issue.

[PromoFaux]

Have another Ubuntu VM

Is this also on the proxmox host?

[andrasg]

Oddly, it looks like the container is not even created properly:

andrasg@ubuntu-sz:~$ docker run -d --dns=127.0.0.1 -e PIHOLE_DNS_=192.168.1.253 -e PH_VERBOSE=1 --name=piholetest pihole/pihole:2022.09.4
WARNING: Localhost DNS setting (--dns=127.0.0.1) may fail in containers.
6537622f00e678ffe6c0b183f6dc887f1938e0393696b5587c1bea3d0e21c8c0
andrasg@ubuntu-sz:~$ docker logs piholetest
Error response from daemon: readlink /var/lib/docker/overlay2/l: invalid argument
andrasg@ubuntu-sz:~$ docker inspect piholetest
[]
Error response from daemon: readlink /var/lib/docker/overlay2/l: invalid argument

Let me try to get some docker service logs from the docker host (Ubuntu VM)...

The working instance is on a different proxmox host and a different Ubuntu VM.

[github-actions[bot]]

This issue is stale because it has been open 30 days with no activity. Please comment or update this issue or it will be closed in 5 days.