core
core copied to clipboard
opnsense
Suricata huge log files fill up the drive and crash the system
Important notices
Before you add a new report, we ask you kindly to acknowledge the following:
- [x] I have read the contributing guide lines at https://github.com/opnsense/core/blob/master/CONTRIBUTING.md
- [x] I am convinced that my issue is new after having checked both open and closed issues at https://github.com/opnsense/core/issues?q=is%3Aissue
When using IDS/IPS is enabled, the suricata log files grow larger beyond an expected size and eventually fill the system drive which causes the system to crash. In the first case (on OPNsense 23.1 Community Edition in June of 2023) the files grew so much over night, that I got from about 110 GB in the evening to 4.4TB (as shown in the screenshots) in the next morning. I had to mount the zfs pool on another device to be able to delete the log files and make OPNsense boot again. I didn't submit any issue report back then, as the issue seemed to be solved in OPNsense Version 23.7. Now it happened again on the 1st February 2024. I'm now on OPNsense 23.10.2 Enterprise Edition. The stream of events is quite similar to the previous one. The /var/log/ directory contained about 70GB of log files, 78% of which where in the ./suricata fiolder. In the next morning the drive was one again filled up and the system had partially crashed. This time I was able to delete the files without pulling the drive.
I have posted my first experience on the forum and there seems to be another user affected by the same problem. As the problem exists on the enterprise channel as well as the community one, I assume that this has to do with the suricata version used in both cases.
To Reproduce
Steps to reproduce the behavior:
- Enable intrusion detection and intrusion prevention on the WAN Interface
- Set the scan mode to 'Hyperscan'
- Download and enable the rules
- Wait for the logs to grow
Expected behavior
The amount/size of log files should be less than the maximum size of the drive and shouldn't be in the terabyte range.
Describe alternatives you considered
As a short term alternative I figured to turn down the amount of saved protocols, but that doesn't help long term. If the problem doesn't get fixed in the foreseeable future, I have to switch to a different solution by another vendor.
Screenshots
If applicable, add screenshots to help explain your problem.
/var/log/suricata/*
If applicable, information from log files supporting your claim.
Unfortunately the majority of log files couldn't be opened due to space constraints and the ones that could be opened didn't contain any valuable information
Add any other context about the problem here.
I run multiple public-facing services and a home lab with two servers. That's my reason for using OPNsense in the first place. So this could be the result of a dos attack, but the log don't provide enough evidence to back that assumption.
Post on the OPNsense Forum: https://forum.opnsense.org/index.php?topic=34489.msg167072#msg167072
Environment
Software version used and hardware type if relevant, e.g.:
System of first occurence:
OPNsense 23.7 (amd64) Intel Xeon E3-1225 V2 Realtek Gigabit Network Adapter (I don't know the model unfortunately)
System now:
OPNsense 23.10.2 (amd64). Deciso DEC2750 Rack Security Appliance
If I'm not mistaken, we recently added https://github.com/opnsense/core/commit/528b7df875560f572a087f32e6322e75edbc3a9c to our business edition. If that's the case, you should be able to also add size limits for log files (in System: Settings: Logging).
Thank you for pointing that out! I never head of that setting before. But I still would like to find out why suricata writes that much log, whether it's a misconfiguration or a bug in suricata.
The log cleanup feature was added recently for situations like these as a result of external testing, filter logs can grow quite large as well in which case high traffic sites either want to consider external logging only or limiting storage for hotspots like these.
You should be able to inspect the contents of these files via Services: Intrusion Detection: Log File or just crawl through it with command line tools like grep or less. If the eve files are small and the suricata log is quite large, I would expect a misconfiguration, you can show the last 100 lines from the current log using: tail -n 100 /var/log/suricata/latest.log.
This issue has been automatically timed-out (after 180 days of inactivity).
For more information about the policies for this repository, please read https://github.com/opnsense/core/blob/master/CONTRIBUTING.md for further details.
If someone wants to step up and work on this issue, just let us know, so we can reopen the issue and assign an owner to it.
