Gen_Stats Broken Condition

[vJoseph]

When I enable 'Extra DHCP Logging' from the DHCP and DNS/Logging tab in Luci, the 'gen_stats' command doesn't work. No error gets kicked out and everything seems like it is working but the logread only shows 'generating stats' string and doesn't show the usual lines of stats afterwards.

[lynxthecat]

@dave14305 any clue why the --log-dhcp option of dnsmasq seems here to prevent the USR1 triggered stats from dnsmasq from showing up?

https://openwrt.org/docs/guide-user/base-system/dhcp#all_options

https://thekelleys.org.uk/dnsmasq/docs/dnsmasq-man.html

[dave14305]

There’s no obvious connection between the two. Have you confirmed the behavior?

[lynxthecat]

No I haven't. @vJoseph can you try manually sending the USR1 signal to the dnsmasq PID using:

kill -USR1 X

where X is the process ID that can be found using ps?

[dave14305]

It should be done in the same way the script does it:

kill -USR1 "$(pidof /usr/sbin/dnsmasq)"

[dave14305]

@friendly-bits What are the probabilities of this failing and exiting? https://github.com/lynxthecat/adblock-lean/blob/40483b52bb7397187d3142e236d339e9eca4ec30/adblock-lean#L1001

[friendly-bits]

@friendly-bits What are the probabilities of this failing and exiting?

reg_action() basically does 2 things: calls log_msg() (return code is not checked) and when performing actions which require lock, it will update the pid file with the action which is being executed via a call to update_pid_action() (return code is checked).

reg_action() will only return 1 when both conditions are satisfied:

1. ${LOCK_REQ} is set (it is not set for action gen_stats)
2. update_pid_action() returns 1 (which can only happen if there is an issue with acquiring lock or writing to the pid file, but when ${LOCK_REQ} is not set, update_pid_action() is not called at all)

So I think the probability is 0.

[vJoseph]

It should be done in the same way the script does it:

kill -USR1 "$(pidof /usr/sbin/dnsmasq)"

Would you mind explaining what this line does? Just so I know what I'm supposed to be looking for.

Nothing failed, or seemed to, based off my log output. The stats just didn't print to the log. After I remove the extra DHCP Logging option everything works as normal again.

root@OpenWrt:~# service adblock-lean gen_stats
Generating dnsmasq stats.
dnsmasq stats available for reading using 'logread'.
root@OpenWrt:~# logread
Fri Mar 21 16:46:31 2025 user.info adblock-lean: Generating dnsmasq stats.

[friendly-bits]

Would you mind explaining what this line does?

This gets the PID's (process id's) of the running dnsmasq processes, and sends a signal to those processes. The signal is USR1, I.e. user-defined signal (rather than system-defined signal). The 'user' who defines what the signal means for this matter is the developer of dnsmasq. Specifically, what USR1 means for dnsmasq is: print some information to the system log. So this command is what actually triggers dnsmasq to print that info, and you can issue it exactly the same way as adblock-lean does. Which is why @lynxthecat asked you to issue it manually while the DHCP logging option is enabled, and see whether the log entries show up. If they do then something in adblock-lean is not working. If they don't then it's probably an issue with something else.

Also please post the output of following commands (redact sensitive information if any)

cat /etc/config/dhcp
echo "pgrep:"; pgrep -la /usr/sbin/dnsmasq
echo "pidof:"; pidof /usr/sbin/dnsmasq

[vJoseph]

kill -USR1 "$(pidof /usr/sbin/dnsmasq)"

@lynxthecat This command did not do anything for me. Even when I manually put in the PID. I'm assuming it's a DNSMASQ issue now

config dnsmasq
        option domainneeded '1'
        option localise_queries '1'
        option rebind_protection '1'
        option rebind_localhost '1'
        option local '/lan/'
        option domain 'lan'
        option expandhosts '1'
        option cachesize '2000'
        option authoritative '1'
        option readethers '1'
        option leasefile '/tmp/dhcp.leases'
        option localservice '1'
        option ednspacket_max '1232'
        list addnmount '/bin/busybox'
        list server '127.0.0.1#5453'
        option noresolv '1'
        option logdhcp '1'

config dhcp 'lan'
        option interface 'lan'
        option start '100'
        option limit '150'
        option leasetime '12h'
        option dhcpv4 'server'
        list dhcp_option '42,192.168.1.1'

config dhcp 'wan'
        option interface 'wan'
        option ignore '1'

config odhcpd 'odhcpd'
        option maindhcp '0'
        option leasefile '/tmp/hosts/odhcpd'
        option leasetrigger '/usr/sbin/odhcpd-update'
        option loglevel '4'

root@OpenWrt:~# echo "pgrep:"; pgrep -la /usr/sbin/dnsmasq
pgrep:
22569 /usr/sbin/dnsmasq

root@OpenWrt:~# echo "pidof:"; pidof /usr/sbin/dnsmasq
pidof:
22569

[dave14305]

I just tested the kill with and without the logdhcp option and both work. Something else must also be changing in the dnsmasq config during the restart, or else something is wrong in adblock-lean.

[friendly-bits]

kill -USR1 "$(pidof /usr/sbin/dnsmasq)"

Just to make sure: did you check the output of logread after manually running this command?

[vJoseph]

@friendly-bits yes I did a logread command after the USR1. Nothing printed

I had a suspicion that my DNSMASQ may have been an issue. I am using Stubby for DoT, other then configuring it for that my DNSMASQ is pretty much default settings, without ipv6 enabled . It's just weird that only the extra DHCP Logging option breaks it

[friendly-bits]

To rule out (or in?) adblock-lean, you can run the command service adblock-lean stop, make sure that extra DHCP logging is enabled, then manually issue the command kill -USR1 "$(pidof /usr/sbin/dnsmasq)" and check in logread. If the info is there then adblock-lean is somehow affecting this. If the info is not there then this has nothing to do with adblock-lean.

Also what is the output of ubus call system board?

[vJoseph]

{ "kernel": "5.15.162", "hostname": "OpenWrt", "system": "ARMv8 Processor rev 4", "model": "FriendlyElec NanoPi R4S", "board_name": "friendlyarm,nanopi-r4s", "rootfs_type": "ext4", "release": { "distribution": "OpenWrt", "version": "23.05.4", "revision": "r24012-d8dd03c46f", "target": "rockchip/armv8", "description": "OpenWrt 23.05.4 r24012-d8dd03c46f" } }

When I stopped adblock-lean dnsmasq DID print out the stats correctly, with extra DHCP Logging enabled.

I restarted adblock-lean and used the USR1 command again and it actually worked.. just that first time and then it stopped working again.

[friendly-bits]

@vJoseph it could be the case that this command stops responding some time after restarting the dnsmasq service because of some weird interaction. Both stopping and (re)starting adblock-lean involves stopping and starting the dnsmasq service, so to me it is still unclear whether adblock-lean is causing this or not. To verify that this is actually caused by adblock-lean, please stop adblock-lean, check whether the command responds, wait for a bit (a couple minutes, just to be sure) and then check the command again.

[vJoseph]

@friendly-bits I have had adblock-lean stopped for the past 8 hours and I am still able to generate DNSMasq stats with USR1 without issues.. I've also realized that my log is actually showing the extra DHCP Logging info now.

In conclusion, in addition to the gen_stats being broken, the entirety of my DHCP logging is broken whenever I have extra DHCP Logging enabled & adblock-lean running.

This isn't a huge issue since I'm not going to be running extra DHCP logging indefinitely, I just wanted to check some stuff with my setup.

I appreciate your time on helping with this phenomenon

[friendly-bits]

Thing is: adblock-lean does not actually "run" after loading the blocklist into dnsmasq. It's just dnsmasq running with the loaded blocklist. The way adblock-lean generates the blocklist is unlikely to affect this because the resulting blocklist is 100% conformant to the dnsmasq format (otherwise it wouldn't load at all), and we ensure this 4 times:

1. While processing source lists: by validating that nothing weird is going on in each line
2. By running the command dnsmasq --test on the resulting final processed blocklist
3. By testing dns resolution after loading the blocklist and restarting the dnsmasq service
4. By injecting a test entry into the final blocklist and testing for the presence of that entry via a DNS query after loading the blocklist (which essentially provides a definite answer whether the blocklist was indeed loaded by dnsmasq)

So essentially: loading the blocklist into dnsmasq seems to cause this issue. This is a direct consequence of using adblock-lean which generates and loads the blocklist. However I believe that same thing would happen when loading any other blocklist of comparable size (at least using the method used by adblock-lean - via a conf-file).

So to me this sounds like a bug in dnsmasq, possibly affected by something else in the system. Also I couldn't reproduce the issue on my test system (admittedly it is running a snapshot of OpenWrt, not 23.05.4 as your system), so I don't have a way to troubleshoot this.

[friendly-bits]

One other thing. Could you disable compression (use_compression="0" in adblock-lean config) and then re-test?

[vJoseph]

@friendly-bits ~~you have successfully isolated the issue, I disabled compression.~~

So far, everything is functioning correctly, even with extra DHCP logging, I've generated stats multiple times and will check back after a few hours to see if my logging is still working correctly.

EDIT: dnsmasq logging is back to blank outputs, compression on/off does not trigger the issue

[friendly-bits]

Compression was the only special thing about how adblock-lean loads the blocklist (while this is supported by dnsmasq, I believe that no other adblocker on OpenWrt has a similar feature). Otherwise it is just a file in the dnsmasq work directory ("conf-dir") which dnsmasq automatically picks up and treats as a conf file. So alas, I don't think there is anything at the level of adblock-lean code which we can change here to solve this. And the fact that this can't be reproduced on our test systems is a showstopper for this issue. Maybe if you post a question on the general OpenWrt forum, someone will have an idea about this (perhaps OpenWrt devs who know what exactly enabling that logging option does in regards to dnsmasq).

[friendly-bits]

Anyway, just for extra info: please post your adblock-lean config:

cat /etc/adblock-lean/config

I'll try again to reproduce the issue with your specific blocklist selection. If you are using local allowlist or blocklist files (typically in /etc/adblock-lean/allowlist and /etc/adblock-lean/blocklist), please post the contents of those files as well:

cat /etc/adblock-lean/allowlist
cat /etc/adblock-lean/blocklist

[vJoseph]

Compression was the only special thing about how adblock-lean loads the blocklist (while this is supported by dnsmasq, I believe that no other adblocker on OpenWrt has a similar feature). Otherwise it is just a file in the dnsmasq work directory ("conf-dir") which dnsmasq automatically picks up and treats as a conf file. So alas, I don't think there is anything at the level of adblock-lean code which we can change here to solve this. And the fact that this can't be reproduced on our test systems is a showstopper for this issue. Maybe if you post a question on the general OpenWrt forum, someone will have an idea about this (perhaps OpenWrt devs who know what exactly enabling that logging option does in regards to dnsmasq).

I agree. It does sound more like a DNSMasq issue. Perhaps the updated version fixes something that is causing it on my out dated version.

[vJoseph]

root@OpenWrt:~# cat /etc/adblock-lean/config                                                                                                 
# adblock-lean configuration options
# config_format=v6
#
# values must be enclosed in double-quotes
# custom comments are not preserved after automatic config update

# Whitelist mode: only domains (and their subdomains) included in the allowlist(s) are allowed, all other domains are blocked
# In this mode, if blocklists are used in addition to allowlists, subdomains included in the blocklists will be blocked,
# including subdomains of allowed domains
whitelist_mode="0"

# One or more *raw domain* format blocklist/ipv4 blocklist/allowlist urls separated by spaces
blocklist_urls="https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/pro-onlydomains.txt https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/tif-onlydomains.txt"
blocklist_ipv4_urls=""
allowlist_urls=""

# One or more *dnsmasq format* domain blocklist/ipv4 blocklist/allowlist urls separated by spaces
dnsmasq_blocklist_urls=""
dnsmasq_blocklist_ipv4_urls=""
dnsmasq_allowlist_urls=""

# Path to optional local *raw domain* allowlist/blocklist files in the form:
# site1.com
# site2.com
local_allowlist_path="/etc/adblock-lean/allowlist"
local_blocklist_path="/etc/adblock-lean/blocklist"

# Test domains are automatically querried after loading the blocklist into dnsmasq,
# in order to verify that the blocklist didn't break DNS resolution
# If query for any of the test domains fails, previous blocklist is restored from backup
# If backup doesn't exist, the blocklist is removed and adblock-lean is stopped
# Leaving this empty will disable verification
test_domains="google.com microsoft.com amazon.com"

# List part failed action:
# This option applies to blocklist/allowlist parts which failed to download or couldn't pass validation checks
# SKIP - skip failed blocklist file part and continue blocklist generation
# STOP - stop blocklist generation (and fall back to previous blocklist if available)
list_part_failed_action="SKIP"

# Maximum number of download retries
max_download_retries="3"

# Minimum number of good lines in final postprocessed blocklist
min_good_line_count="330000"

# Mininum number of lines of any individual downloaded part
min_blocklist_part_line_count="1"
min_blocklist_ipv4_part_line_count="1"
min_allowlist_part_line_count="1"

# Maximum size of any individual downloaded blocklist part
max_file_part_size_KB="39000"

# Maximum total size of combined, processed blocklist
max_blocklist_file_size_KB="48000"

# Whether to perform sorting and deduplication of entries (usually doesn't cause much slowdown, uses a bit more memory) - enable (1) or disable (0)
deduplication="1"

# compress final blocklist, intermediate blocklist parts and the backup blocklist to save memory - enable (1) or disable (0)
use_compression="1"

# restart dnsmasq if previous blocklist was extracted and before generation of
# new blocklist thereby to free up memory during generaiton of new blocklist - enable (1) or disable (0)
initial_dnsmasq_restart="0"

# Start delay in seconds when service is started from system boot
boot_start_delay_s="120"

# If a path to custom script is specified and that script defines functions 'report_success()' and 'report_failure()'',
# one of these functions will be executed when adblock-lean completes the execution of some commands,
# with the success or failure message passed in first argument
# report_success() is only executed upon completion of the 'start' command
# Recommended path is '/usr/libexec/abl_custom-script.sh' which the luci app has permission to access
custom_script=""

# Crontab schedule expression for periodic list updates
cron_schedule="0 5 * * *"

# dnsmasq instance and config directory
# normally this should be set automatically by the 'setup' command
DNSMASQ_INSTANCE="cfg01411c"
DNSMASQ_INDEX="0"
DNSMASQ_CONF_D="/tmp/dnsmasq.d"

root@OpenWrt:~# cat /etc/adblock-lean/blocklist
jwplayer.com

[vJoseph]

Discovered a way to duplicate this and determined it has nothing to do with adblock-lean, should I close this?

If I restart the log service (to clear the log) service restart log

The gen_stats (SIGUSR1) command will not output statistics.

If I restart the dnsmasq service after I restart the log service service restart dnsmasq

Then the statistics generation will actually output to the log

[lynxthecat]

I’m curious about that behaviour. By the way @friendly-bits, there is a way to output these stats directly rather than have them output to the system log. I forget exactly how but @dave14305 knows.

[friendly-bits]

I made an attempt to figure out what dnsmasq is actually doing on receipt of USR1 by reading the code, but I failed miserably because the logging logic is extremely convoluted, has lots of moving parts and has stuff like this

https://github.com/imp/dnsmasq/blob/770bce967cfc9967273d0acfb3ea018fb7b17522/src/log.c#L351

(setting a variable to 0, immediately checking if the value is 0, doing something provided that the condition is satisfied - which is always true - , setting same variable to 1 and never checking its value again)

which just stomps me. (for the protocol, I'm just learning to read C code, so I may be missing something obvious)

In general, the code seems to implement error handling for logging, but I suspect that somehow receiving USR1 bypasses most of that, and dnsmasq may (?) be trying to write to file descriptor which no longer exists.

This is my best take which may be competely wrong. Perhaps @dave14305 will have better luck than myself at figuring this out.