Option to preserve % of disk space so that logging does not fills up disk space and causes DoS

[dietersar]

Important notices

Before you add a new report, we ask you kindly to acknowledge the following:

• [x] I have read the contributing guide lines at https://github.com/opnsense/core/blob/master/CONTRIBUTING.md
• [x] I am convinced that my issue is new after having checked both open and closed issues at https://github.com/opnsense/core/issues?q=is%3Aissue

Is your feature request related to a problem? Please describe.

Currently, there is only an option to preserve logs for X amount of days (in System:Settings:Logging)

Yet there is no option to limit logs to a certain amount or maximum amount of disk space, or to preserve X % of disk space to make sure the device keeps on running fine. Now, when the disk of a opnsense firewall is full, the device stops responding to GUI logon attempts, additional services stop responding.

Describe the solution you like

An option to preserve % of disk space. This will be a way to prevent disk space (and hence device) during (logging) Denial of Service scenario's.

Describe alternatives you considered

A remote syslog could be a solution if logs are no longer written to localhost. However, this is not possible within every environment, not everybody has dedicated syslog systems in place. So for those setups it would be interesting to have an option to limit disk usage of log files.

Additional context

Also see https://twitter.com/dietersar/status/1522466241985843205

Thx for considering this

[dietersar]

Thinking a bit further I am wondering if this could be done per file, in combination with all files. For example: if available disk space drops below10%, remove oldest logfiles first If available disk space keeps dropping < 10%, the current logfile oldest entries are discarded/overwritten

[AdSchellevis]

I do think we should keep this simple to be honest, if the issue is that the log can lockup the box, it might be simpler to investigate if/how we can constrain the directory size to a certain limit (with zfs this might be easier than with ufs).

Building a script that drops files on patterns is probably best implemented using a custom script and a cron job, either explained in a document (how to do it) or a plugin offering such feature.

At the end of the day, if logs can grow so large, there often is an underlaying issue or misconfiguration in our experience. With the previous rotating log files these usually where hidden, although it did slow down the machine (a lot).

[dietersar]

There was an issue on the network causing the huge logfile - an entry for not logging dropped packets from a particular host (temporarily) solved this issue for now.

However, I do like the idea to restrict (certain) log directory size(s).

The behavior I observed was that there was no connection anymore, after a reboot the device was functioning - yet only the firewalling part. GUI access was not possible. df -alh even showed a negative value as availability using du -ah I drilled this down to be one of the filter log files (SSH access was possible luckily).

[mimugmail]

Monit should offer a check for folder and you might add a regexp for deleting oldest logs

[ikke-t]

I have now used an hour investigating why my home nework is broken. Turns out opnsense filter logs consumed >5G of space, filling the disk. This lead to dhcp not working among with the other services. But as DHCP didn't work, it messed up things badly. So yes, definately it would be in place to have such a high watermark for logs.

[ikke-t]

here are the logs it creates:

root@OPNsense:~ # ls -lh /var/log/filter/*
-rw-------  1 root  wheel   185M May 26 00:00 /var/log/filter/filter_20220525.log
-rw-------  1 root  wheel   186M May 27 00:00 /var/log/filter/filter_20220526.log
-rw-------  1 root  wheel   186M May 28 00:00 /var/log/filter/filter_20220527.log
-rw-------  1 root  wheel   194M May 29 00:00 /var/log/filter/filter_20220528.log
-rw-------  1 root  wheel   197M May 30 00:00 /var/log/filter/filter_20220529.log
-rw-------  1 root  wheel   178M May 31 00:00 /var/log/filter/filter_20220530.log
-rw-------  1 root  wheel   186M Jun  1 00:00 /var/log/filter/filter_20220531.log
-rw-------  1 root  wheel   186M Jun  2 00:00 /var/log/filter/filter_20220601.log
-rw-------  1 root  wheel   183M Jun  3 00:00 /var/log/filter/filter_20220602.log
-rw-------  1 root  wheel   178M Jun  4 00:00 /var/log/filter/filter_20220603.log
-rw-------  1 root  wheel   176M Jun  5 00:00 /var/log/filter/filter_20220604.log
-rw-------  1 root  wheel   183M Jun  6 00:00 /var/log/filter/filter_20220605.log
-rw-------  1 root  wheel   186M Jun  6 23:59 /var/log/filter/filter_20220606.log
-rw-------  1 root  wheel   187M Jun  8 00:00 /var/log/filter/filter_20220607.log
-rw-------  1 root  wheel   176M Jun  9 00:00 /var/log/filter/filter_20220608.log
-rw-------  1 root  wheel   174M Jun 10 00:00 /var/log/filter/filter_20220609.log
-rw-------  1 root  wheel   172M Jun 11 00:00 /var/log/filter/filter_20220610.log
-rw-------  1 root  wheel   176M Jun 12 00:00 /var/log/filter/filter_20220611.log
-rw-------  1 root  wheel   178M Jun 13 00:00 /var/log/filter/filter_20220612.log
-rw-------  1 root  wheel   185M Jun 14 00:00 /var/log/filter/filter_20220613.log
-rw-------  1 root  wheel   209M Jun 15 00:00 /var/log/filter/filter_20220614.log
-rw-------  1 root  wheel   242M Jun 15 23:59 /var/log/filter/filter_20220615.log
-rw-------  1 root  wheel   213M Jun 17 00:00 /var/log/filter/filter_20220616.log
-rw-------  1 root  wheel   200M Jun 18 00:00 /var/log/filter/filter_20220617.log
-rw-------  1 root  wheel   178M Jun 19 00:00 /var/log/filter/filter_20220618.log
-rw-------  1 root  wheel   188M Jun 20 00:00 /var/log/filter/filter_20220619.log
-rw-------  1 root  wheel   198M Jun 21 00:00 /var/log/filter/filter_20220620.log
-rw-------  1 root  wheel   194M Jun 22 00:00 /var/log/filter/filter_20220621.log
-rw-------  1 root  wheel   194M Jun 23 00:00 /var/log/filter/filter_20220622.log
-rw-------  1 root  wheel   194M Jun 24 00:00 /var/log/filter/filter_20220623.log
-rw-------  1 root  wheel    12M Jun 24 10:11 /var/log/filter/filter_20220624.log
lrwxr-x---  1 root  wheel    35B Jun 24 00:01 /var/log/filter/latest.log -> /var/log/filter/filter_20220624.log

and here is roughly how the log space gets spent:

root@OPNsense:~ # du -sm /var/log/* | sort -n
0       /var/log/lastlog
0       /var/log/radius.log
0       /var/log/radutmp
0       /var/log/radwtmp
0       /var/log/suricata.log.0
0       /var/log/suricata.log.1
0       /var/log/suricata.log.2
0       /var/log/suricata.log.3
0       /var/log/suricata.log.4
0       /var/log/suricata.log.5
0       /var/log/suricata.log.6
1       /var/log/acmeclient
1       /var/log/acmeclient.log
1       /var/log/audit
1       /var/log/audit.log
1       /var/log/bsdinstaller
1       /var/log/configd.log
1       /var/log/dhcpd.log
1       /var/log/dmesg.today
1       /var/log/dmesg.yesterday
1       /var/log/dnsmasq
1       /var/log/dnsmasq.log
1       /var/log/filter.log
1       /var/log/gateways.log
1       /var/log/haproxy.log
1       /var/log/ipfw.today
1       /var/log/ipfw.yesterday
1       /var/log/ipsec.log
1       /var/log/lighttpd
1       /var/log/lighttpd.log
1       /var/log/mount.today
1       /var/log/mount.yesterday
1       /var/log/ntp
1       /var/log/ntpd
1       /var/log/ntpd.log
1       /var/log/openvpn.log
1       /var/log/pf.today
1       /var/log/pf.yesterday
1       /var/log/pkg
1       /var/log/pkg.log
1       /var/log/portalauth
1       /var/log/portalauth.log
1       /var/log/ppps.log
1       /var/log/radacct
1       /var/log/resolver.log
1       /var/log/routing.log
1       /var/log/setuid.today
1       /var/log/setuid.yesterday
1       /var/log/squid.log
1       /var/log/squid.syslog.log
1       /var/log/suricata
1       /var/log/suricata.log
1       /var/log/suricata.syslog.log
1       /var/log/system
1       /var/log/system.log
1       /var/log/userlog
1       /var/log/utx.lastlogin
1       /var/log/utx.log
1       /var/log/vpn.log
1       /var/log/wireless.log
2       /var/log/routing
4       /var/log/configd
5       /var/log/acme.sh.log
7       /var/log/dhcpd
8       /var/log/flowd.log
8       /var/log/haproxy
11      /var/log/flowd.log.000001
11      /var/log/flowd.log.000002
11      /var/log/flowd.log.000003
11      /var/log/flowd.log.000004
11      /var/log/flowd.log.000005
11      /var/log/flowd.log.000006
11      /var/log/flowd.log.000007
11      /var/log/flowd.log.000008
11      /var/log/flowd.log.000009
12      /var/log/flowd.log.000010
22      /var/log/telegraf.log
23      /var/log/telegraf
24      /var/log/openvpn
48      /var/log/squid
5675    /var/log/filter

next thing to figure out why does it log so much, most of them are allowed traffic. I would not need to store allowed traffic for a month. Perhaps there is an option for it.

[ikke-t]

ok, i found the option to limit it to given days. That will help already

[OPNsense-bot]

This issue has been automatically timed-out (after 180 days of inactivity).

For more information about the policies for this repository, please read https://github.com/opnsense/core/blob/master/CONTRIBUTING.md for further details.

If someone wants to step up and work on this issue, just let us know, so we can reopen the issue and assign an owner to it.

[nunofernandes]

@ikke-t , where did you find that option?

Thanks

[dietersar]

@ikke-t , where did you find that option?

Thanks

That is in System / Settings / Logging