docker-pi-hole
docker-pi-hole copied to clipboard
pi-hole
Logrotate
ideas:
- check if log is listed in var/lib/logrotate/status
- according to rotation (daily, weekly, monthly), log creation date is ok.
https://serverfault.com/questions/189320/how-can-i-monitor-what-logrotate-is-doing
So I suppose mine is a known issue... I've just realised my pihole.log is 8GB and my FTL.log is 70MB. Both stopped rotating back in August 2022. I'm running release 2024.02.0 on Synology DSM 7.1.1-42962 Update 6.
Here is what I've tried:
#docker exec -it pihole1 bash
root@pihole1:/# ls -lah /var/log/pihole
total 8.1G
drwx------ 1 www-data 999 290 Aug 25 2022 .
drwxr-xr-x 1 root root 174 Feb 20 22:33 ..
-rw-r--r-- 1 www-data pihole 70M Feb 20 22:42 FTL.log
-rwx------ 1 root root 14K Aug 25 2022 FTL.log.1
-rwx------ 1 root root 2.5K Aug 24 2022 FTL.log.2.gz
-rwx------ 1 root root 20K Aug 23 2022 FTL.log.3.gz
-rwx------ 1 root root 19M Jul 9 2022 pihole-FTL.log
-rw-r--r-- 1 www-data pihole 8.0G Feb 20 22:49 pihole.log
-rwx------ 1 root root 9.3M Aug 25 2022 pihole.log.1
-rwx------ 1 root root 849K Aug 24 2022 pihole.log.2.gz
-rwx------ 1 root root 703K Aug 23 2022 pihole.log.3.gz
-rwx------ 1 root root 880K Aug 22 2022 pihole.log.4.gz
-rwx------ 1 www-data root 8.4K Feb 20 22:05 pihole_updateGravity.log
root@pihole1:/# pihole flush once
[i] Flushing /var/log/pihole/pihole.log ...error: Ignoring /etc/pihole/logrotate because the file owner is wrong (should be root or user with uid 0).
[✓] Flushed /var/log/pihole/pihole.log
[✓] Deleted queries from database
I've tried to force root ownership of the log files:
root@pihole1:/# chown root:root /var/log/pihole/pihole.log /var/log/pihole/pihole-FTL.log
root@pihole1:/# ls -lah /var/log/pihole
total 8.1G
drwx------ 1 www-data 999 290 Aug 25 2022 .
drwxr-xr-x 1 root root 174 Feb 20 22:33 ..
-rw-r--r-- 1 www-data pihole 70M Feb 20 22:50 FTL.log
-rwx------ 1 root root 14K Aug 25 2022 FTL.log.1
-rwx------ 1 root root 2.5K Aug 24 2022 FTL.log.2.gz
-rwx------ 1 root root 20K Aug 23 2022 FTL.log.3.gz
-rwx------ 1 root root 19M Jul 9 2022 pihole-FTL.log
-rw-r--r-- 1 root root 8.0G Feb 20 22:54 pihole.log
-rwx------ 1 root root 9.3M Aug 25 2022 pihole.log.1
-rwx------ 1 root root 849K Aug 24 2022 pihole.log.2.gz
-rwx------ 1 root root 703K Aug 23 2022 pihole.log.3.gz
-rwx------ 1 root root 880K Aug 22 2022 pihole.log.4.gz
-rwx------ 1 www-data root 8.4K Feb 20 22:05 pihole_updateGravity.log
root@pihole1:/# pihole flush once
[i] Flushing /var/log/pihole/pihole.log ...error: Ignoring /etc/pihole/logrotate because the file owner is wrong (should be root or user with uid 0).
[✓] Flushed /var/log/pihole/pihole.log
[✓] Deleted queries from database
If I restart the container, the ownership of the pihole.log is restored to the previous values:
#docker restart pihole1
pihole1
# docker exec -it pihole1 bash
root@pihole1:/# ls -lah /var/log/pihole
total 8.1G
drwx------ 1 www-data 999 290 Aug 25 2022 .
drwxr-xr-x 1 root root 174 Feb 20 22:33 ..
-rw-r--r-- 1 www-data pihole 70M Feb 20 22:55 FTL.log
-rwx------ 1 root root 14K Aug 25 2022 FTL.log.1
-rwx------ 1 root root 2.5K Aug 24 2022 FTL.log.2.gz
-rwx------ 1 root root 20K Aug 23 2022 FTL.log.3.gz
-rwx------ 1 root root 19M Jul 9 2022 pihole-FTL.log
-rw-r--r-- 1 www-data pihole 8.0G Feb 20 22:55 pihole.log
-rwx------ 1 root root 9.3M Aug 25 2022 pihole.log.1
-rwx------ 1 root root 849K Aug 24 2022 pihole.log.2.gz
-rwx------ 1 root root 703K Aug 23 2022 pihole.log.3.gz
-rwx------ 1 root root 880K Aug 22 2022 pihole.log.4.gz
-rwx------ 1 www-data root 8.4K Feb 20 22:05 pihole_updateGravity.log
I'm running pihole using the following docker-compose.yaml
version: '3.7'
networks:
# references externally defined network host_lan, see docker-create-host_lan.sh
host_lan:
external: true
services:
pihole:
container_name: ${PIHOLE_CONTAINER_NAME1}
image: ${PIHOLE_IMG_NAME}:${TAG:-latest}
hostname: ${PIHOLE_HOSTNAME1}
domainname: ${DOMAIN_NAME}
networks:
# connect to host_lan with static IP address
host_lan:
ipv4_address: ${IPV4_PIHOLE1}
# set DNS servers required by pihole
dns:
- 127.0.0.1
- 8.8.8.8
# set environment as required by pihole
env_file: pihole.env
environment:
FTLCONF_LOCAL_IPV4: ${IPV4_PIHOLE1}
volumes:
# set volume binding as required by pihole
- ${HOST_DOCKER_DIR}/${PIHOLE_CONTAINER_NAME1}/etc/pihole:/etc/pihole
- ${HOST_DOCKER_DIR}/${PIHOLE_CONTAINER_NAME1}/etc/dnsmasq.d:/etc/dnsmasq.d
- ${HOST_DOCKER_DIR}/${PIHOLE_CONTAINER_NAME1}/log/:/var/log/pihole
restart: unless-stopped
healthcheck:
disable: true
and this is my pihole.env
WEBPASSWORD=xxxxxxx
TZ=Europe/Rome
VIRTUAL_HOST=pihole.mydomain.com
REV_SERVER=true
REV_SERVER_TARGET=192.168.1.80
REV_SERVER_DOMAIN=mydomain.com
REV_SERVER_CIDR=192.168.1.1/24
PIHOLE_DNS_=192.168.1.80
WEBTHEME=default-auto
PIHOLE_UID=1041
PIHOLE_GID=100
WEB_UID=1041
WEB_GID=100
Do you see the same behavior if you remove WEB_UID and WEB_GID variables?
@rdwebdesign yes, even if I remove those variables and recreate the container, I see almost the same behaviour. The ownership of the log files differs, as follows:
root@pihole1:/# ls -lah /var/log/pihole
total 8.1G
drwx------ 1 pihole 999 290 Aug 25 2022 .
drwxr-xr-x 1 root root 174 Feb 21 20:08 ..
-rw-r--r-- 1 pihole users 70M Feb 21 20:08 FTL.log
-rwx------ 1 root root 14K Aug 25 2022 FTL.log.1
-rwx------ 1 root root 2.5K Aug 24 2022 FTL.log.2.gz
-rwx------ 1 root root 20K Aug 23 2022 FTL.log.3.gz
-rwx------ 1 root root 19M Jul 9 2022 pihole-FTL.log
-rw-r--r-- 1 pihole users 8.0G Feb 21 20:08 pihole.log
-rwx------ 1 root root 9.3M Aug 25 2022 pihole.log.1
-rwx------ 1 root root 849K Aug 24 2022 pihole.log.2.gz
-rwx------ 1 root root 703K Aug 23 2022 pihole.log.3.gz
-rwx------ 1 root root 880K Aug 22 2022 pihole.log.4.gz
-rwx------ 1 pihole root 8.4K Feb 20 22:05 pihole_updateGravity.log
instead, here is the ownership with the two variables set:
root@pihole1:/# ls -lah /var/log/pihole
total 8.1G
drwx------ 1 www-data 999 290 Aug 25 2022 .
drwxr-xr-x 1 root root 174 Feb 21 20:10 ..
-rw-r--r-- 1 www-data pihole 70M Feb 21 20:12 FTL.log
-rwx------ 1 root root 14K Aug 25 2022 FTL.log.1
-rwx------ 1 root root 2.5K Aug 24 2022 FTL.log.2.gz
-rwx------ 1 root root 20K Aug 23 2022 FTL.log.3.gz
-rwx------ 1 root root 19M Jul 9 2022 pihole-FTL.log
-rw-r--r-- 1 www-data pihole 8.0G Feb 21 20:16 pihole.log
-rwx------ 1 root root 9.3M Aug 25 2022 pihole.log.1
-rwx------ 1 root root 849K Aug 24 2022 pihole.log.2.gz
-rwx------ 1 root root 703K Aug 23 2022 pihole.log.3.gz
-rwx------ 1 root root 880K Aug 22 2022 pihole.log.4.gz
-rwx------ 1 www-data root 8.4K Feb 20 22:05 pihole_updateGravity.log
In both cases pihole flush once command fails as follows:
root@pihole1:/# pihole flush once
[i] Flushing /var/log/pihole/pihole.log ...error: Ignoring /etc/pihole/logrotate because it is writable by group or others.
[✓] Flushed /var/log/pihole/pihole.log
[✓] Deleted queries from database
I went to "mirror" my PiHole to another server and I saw that my pihole.log was over 11GB!
There is a logrotate file in my "pihole" storage directory and it contains the right data to logrotate, but doesn't appear to run. There are 6 files in the /etc/logrotate.d directory: alternatives, apt, btmp, dpkg, lighttpd, wtmp. But no pihole. I copied the contents of these files, created a new directory to store these files, copied the files there, then also copied the logrotate file.
Initially, I got an error, but it wanted the files to be "root" created so I did that.
Then I tested logrotate with command logrotate -d /etc/logrotate.d/logrotate. It appeared to work!
Hope this helps someone...
Update
Another error I got was that there is no /var/lib/logrotate/status file. The other thing I noticed is that there is not a crontab setup for the root user. But, idk how logrotate can be called in other ways.
Have just checked my running v6 container and logrotate certainly appears to be working correctly here:
dev-v6:/# ls -lah /var/log/pihole
total 9.0M
drwxr-xr-x 1 pihole pihole 348 Mar 29 00:00 .
drwxr-xr-x 1 root root 60 Mar 15 21:30 ..
-rw-r----- 1 pihole pihole 14K Mar 29 21:57 FTL.log
-rw-r----- 1 pihole pihole 15K Mar 29 00:00 FTL.log.1
-rw-r----- 1 pihole pihole 1.4K Mar 28 00:00 FTL.log.2.gz
-rw-r----- 1 pihole pihole 1.3K Mar 27 00:00 FTL.log.3.gz
-rw-r----- 1 pihole pihole 4.8M Mar 29 21:57 pihole.log
-rw-r----- 1 pihole pihole 3.1M Mar 29 00:00 pihole.log.1
-rw-r----- 1 pihole pihole 413K Mar 28 00:00 pihole.log.2.gz
-rw-r----- 1 pihole pihole 286K Mar 27 00:00 pihole.log.3.gz
-rw-r----- 1 pihole pihole 236K Mar 26 00:00 pihole.log.4.gz
-rw-r----- 1 pihole pihole 205K Mar 25 00:00 pihole.log.5.gz
-rw-r--r-- 1 root root 1.2K Mar 23 03:25 pihole_updateGravity.log
-rw-r----- 1 pihole pihole 0 Mar 16 00:00 webserver.log
-rw-r----- 1 pihole pihole 424 Mar 16 00:00 webserver.log.1
Which is fired nightly by cron
dev-v6:/# cat /etc/crontabs/root
25 3 * * 6 PATH="$PATH:/usr/sbin:/usr/local/bin/" pihole updateGravity >/var/log/pihole/pihole_updateGravity.log || cat /var/log/pihole/pihole_updateGravity.log
00 00 * * * PATH="$PATH:/usr/sbin:/usr/local/bin/" pihole flush once quiet
3 19 * * * PATH="$PATH:/usr/sbin:/usr/local/bin/" pihole updatechecker
Looking at the command that is run (pihole flush once quiet) we see this code:
And the same in my v5 container:
root@docker-pihole-pi4:/# ls -lah /var/log/pihole
total 158M
drwxr-xr-x 1 root root 4.0K Mar 29 21:55 .
drwxr-xr-x 1 root root 4.0K Mar 1 06:44 ..
-rw-r--r-- 1 pihole pihole 216K Mar 29 21:58 FTL.log
-rw-r--r-- 1 pihole pihole 99M Mar 29 21:55 FTL.log.1
-rw-r--r-- 1 pihole pihole 7.5M Mar 29 00:00 FTL.log.2.gz
-rw-r--r-- 1 pihole pihole 8.8M Mar 28 00:00 FTL.log.3.gz
-rw-r--r-- 1 pihole pihole 75K Mar 29 21:58 pihole.log
-rw-r--r-- 1 pihole pihole 33M Mar 29 21:55 pihole.log.1
-rw-r--r-- 1 pihole pihole 2.2M Mar 29 00:00 pihole.log.2.gz
-rw-r--r-- 1 pihole pihole 2.5M Mar 28 00:00 pihole.log.3.gz
-rw-r--r-- 1 pihole pihole 2.9M Mar 27 00:00 pihole.log.4.gz
-rw-r--r-- 1 pihole pihole 2.9M Mar 26 00:00 pihole.log.5.gz
-rw-r----- 1 root pihole 34K Jan 3 17:46 pihole_debug.log
-rw-r--r-- 1 root root 1.2K Mar 24 04:33 pihole_updateGravity.log
Does anyone in this thread still have problems with the logs not being rotated?
@PromoFaux yes, the problems I reported earlier are still there:
$ sudo docker exec -it pihole1 bash
root@pihole1:/# pihole flush once
[i] Flushing /var/log/pihole/pihole.log ...error: Ignoring /etc/pihole/logrotate because the file owner is wrong (should be root or user with uid 0).
[✓] Flushed /var/log/pihole/pihole.log
[✓] Deleted queries from database
root@pihole1:/#
exit
$ ls -lah /volume1/docker/pihole1/log/
total 9.6G
drwxrwxrwx+ 1 pihole synopkgs 290 Aug 25 2022 .
drwxrwxrwx+ 1 pihole users 12 Jan 26 2020 ..
-rw-r--r-- 1 pihole 1000 96M May 8 22:34 FTL.log
-rwxrwxrwx+ 1 root root 14K Aug 25 2022 FTL.log.1
-rwxrwxrwx+ 1 root root 2.5K Aug 24 2022 FTL.log.2.gz
-rwxrwxrwx+ 1 root root 20K Aug 23 2022 FTL.log.3.gz
-rwxrwxrwx+ 1 root root 19M Jul 9 2022 pihole-FTL.log
-rw-r--r-- 1 pihole 1000 9.5G May 8 22:34 pihole.log
-rwxrwxrwx+ 1 root root 9.3M Aug 25 2022 pihole.log.1
-rwxrwxrwx+ 1 root root 849K Aug 24 2022 pihole.log.2.gz
-rwxrwxrwx+ 1 root root 703K Aug 23 2022 pihole.log.3.gz
-rwxrwxrwx+ 1 root root 880K Aug 22 2022 pihole.log.4.gz
-rwxrwxrwx+ 1 pihole root 7.5K May 5 03:42 pihole_updateGravity.log
What does your compose file look like? (Also, have you tried the beta v6 image?)
my compose file:
version: '3.7'
networks:
# references externally defined network host_lan, see docker-create-host_lan.sh
host_lan:
external: true
services:
pihole:
container_name: ${PIHOLE_CONTAINER_NAME1}
image: ${PIHOLE_IMG_NAME}:${TAG:-latest}
hostname: ${PIHOLE_HOSTNAME1}
domainname: ${DOMAIN_NAME}
networks:
# connect to host_lan with static IP address
host_lan:
ipv4_address: ${IPV4_PIHOLE1}
# set DNS servers required by pihole
dns:
- 127.0.0.1
- 8.8.8.8
# set environment as required by pihole
env_file: pihole.env
environment:
FTLCONF_LOCAL_IPV4: ${IPV4_PIHOLE1}
volumes:
# set volume binding as required by pihole
- ${HOST_DOCKER_DIR}/${PIHOLE_CONTAINER_NAME1}/etc/pihole:/etc/pihole
- ${HOST_DOCKER_DIR}/${PIHOLE_CONTAINER_NAME1}/etc/dnsmasq.d:/etc/dnsmasq.d
- ${HOST_DOCKER_DIR}/${PIHOLE_CONTAINER_NAME1}/log/:/var/log/pihole
restart: unless-stopped
healthcheck:
disable: true
my pihole.env:
WEBPASSWORD=xxxxxxx
TZ=Europe/Rome
VIRTUAL_HOST=pihole.mydomain.com
REV_SERVER=true
REV_SERVER_TARGET=192.168.1.80
REV_SERVER_DOMAIN=mydomain.com
REV_SERVER_CIDR=192.168.1.1/24
PIHOLE_DNS_=192.168.1.80
WEBTHEME=default-auto
PIHOLE_UID=1041
PIHOLE_GID=100
WEB_UID=1041
WEB_GID=100
No, I haven't tried the beta v6 image.
PIHOLE_UID=1041 PIHOLE_GID=100 WEB_UID=1041 WEB_GID=100
Here in lies the root cause of the issue.
See the Experimental Variables section of the README file. PIHOLE_UID and WEB_UIID must not be the same
Thank you, I hadn't seen that...
But now I'm going through permission hell...
If I provide no UID and no GID variable, FTL won't start.
If I provide only PIHOLE_UID and PIHOLE_UID (matching a specific user and group created on my Synology NAS host) but no WEB_?ID variable, then the web interface doesn't show anything meaningful apart from permission errors in the settings page.
But then should I create also a user on my Synology NAS for the web interface? Does the setup need to be so complex or is anything wrong with mine? Confused....
Ah, Synology NAS:
https://github.com/pi-hole/docker-pi-hole/issues/963
Actually, at this stage I'd recommend using the v6 beta image instead as it's fairly "close" to release, which removes the WEB_?ID variables as the web page is now served by pihole-FTL itself.
I'm currently running the v6 images on my Synology with DNSMASQ_USER set to pihole (which is the default anyway) and as I'm using a named volume rather than a host mount - I haven't needed to set the pihole_?id variables
Take a read over the new Readme first, though, as you'll need to make a few changes to your environment variables
Thank you for pointing that out.
For the moment I reverted back to my previous situation, i.e. keeping my UID's and GID's the same (although it's wrong) and also stick to image 2024.03.2, since also upgrading to 2024.05.0 by itself seemed to disrupt my pihole (maybe because of my wrong variables).
Now at least I've got my pihole working. I've lived with my huge log file for quite a while, so I'll wait a bit more for v6 to be released, since it's fairly close. Then I'll try to upgrade and hopefully fix the logrotate issue.
Things seem pretty messy with my current set up, I'd rather avoid adding potential new issues due to beta releases at this point.
Going back to the original issue raised, what should happen if the system detects that logrotate isn’t running properly?
It's a good question - I'm not 100% sure there is an answer to it. I suppose we could set a cron script to periodically check the size of the logs - but this does not seem like an ideal solution.
I created this issue before I'd really looked into whether or not logrotate was working in the image... as it happens, it appears to be. Do you have any suggestions?
How do you generate messages like this?
Maybe at the same time as you check the CPU load you can check the log size and/or age of last archived log. Then generate a message accordingly.
The more I think of it, with a properly configured container - there shouldn't be an issue with logrotate.
That diagnosis table is populated by FTL - not sure it would be a good idea for FTL to monitor the size of logs / log rotation status. Certainly on size, what is large for one user is tiny for another.
@buxm's issue appears to be caused by some permission issues on the filesystem - and I've a feeling it would be sorted out with the v6 image in any case
It's a good question - I'm not 100% sure there is an answer to it. I suppose we could set a cron script to periodically check the size of the logs - but this does not seem like an ideal solution.
I created this issue before I'd really looked into whether or not logrotate was working in the image... as it happens, it appears to be. Do you have any suggestions?
My initial thought was a script to check that the logrotate location was writable by the proper user, write something to the log, force a rotate, and check it's non-zero. If any of those fail, error and alert the user somehow.
The more I think about it though, the less I think anything should be done. This is a giant container doing a lot of things. If logrotate works in the base image, no need to overcomplicate its setup and end up scaring off a newcomer who can't diagnose what logrotate is and why it might be throwing an error.