py-idstools icon indicating copy to clipboard operation
py-idstools copied to clipboard

Published 20 hours ago • verified publisher icon jasonish

memory usage increase issue

Open ganduulgag opened this issue 2 years ago • 3 comments

hi,

is there someone who experienced a memory usage increase issue when you run rulecat? if yes, how did you tackle the issue?

in our case, we configured rulecat to run once a day in cron job as we use Security Onion. Whenever the cron job starts, memory usage increases drastically. We suppressed several rules with idstools. As a solution, we added swap space for the moment but we are concerned if this memory usage keeps increasing but we don't want to allocate more swap or memory.

ganduulgag avatar May 20 '22 10:05 ganduulgag

I just looked at rulecat with the ET/Open ruleset and it did use about 500MB. Not ideal, but it does load, parse and sort all the rules in memory. But this might not be the only thing using memory. rulecat can be configured to test the output with suricata -T. This causes suricata to load the rules in memory to check for sanity which can use even more memory. This is somewhat of a known issue, but not much can be done about it if you want to validate the rules before restarted Suricata.

jasonish avatar May 20 '22 15:05 jasonish

where can I configure to test output with suricata -T? Currently, I am using rulecat with ETPro rules and when I run the rulecat, memory usage increases about 3GB which is quite a lot memory for the system. is there a way I can reduce memory usage? besides, it takes roughly 3 min to finish the rulecat. Here is my console output after running rulecat:

2022-09-07 04:05:02,669 - <INFO> -- Loading ./rulecat.conf. 2022-09-07 04:05:02,682 - <INFO> -- Forcing Suricata version to 6.0. 2022-09-07 04:05:02,695 - <INFO> -- Fetching https://urlhaus.abuse.ch/downloads/urlhaus_suricata.tar.gz. 100% - 741376/741376 2022-09-07 04:05:02,934 - <INFO> -- Done. 2022-09-07 04:05:03,126 - <INFO> -- Fetching https://rules.emergingthreatspro.com/0148694801847852/suricata-6.0.0/etpro.rules.tar.gz. 100% - 8922965/8922965 2022-09-07 04:05:04,836 - <INFO> -- Done. 2022-09-07 04:05:05,259 - <INFO> -- Fetching https://sslbl.abuse.ch/blacklist/sslipblacklist.rules. 100% - 14294/14294 2022-09-07 04:05:05,401 - <INFO> -- Done. 2022-09-07 04:05:05,403 - <INFO> -- Fetching https://threatfox.abuse.ch/downloads/threatfox_suricata.tar.gz. 100% - 1073155/1073155 2022-09-07 04:05:05,908 - <INFO> -- Done. 2022-09-07 04:05:06,098 - <INFO> -- Ignoring file rules/deleted.rules 2022-09-07 04:05:57,472 - <INFO> -- Loaded 155162 rules. 2022-09-07 04:08:32,612 - <INFO> -- Disabled 1233 rules. 2022-09-07 04:08:32,612 - <INFO> -- Enabled 0 rules. 2022-09-07 04:08:32,612 - <INFO> -- Modified 19831 rules. 2022-09-07 04:08:32,612 - <INFO> -- Dropped 0 rules. 2022-09-07 04:08:35,223 - <INFO> -- Enabled 184 rules for flowbit dependencies. 2022-09-07 04:09:10,379 - <INFO> -- Writing rules to /home/kali/all.rules: total: 155162; enabled: 137950; added: 13435; removed 11563; modified: 1359 2022-09-07 04:09:13,878 - <INFO> -- Done

ganduulgag avatar Sep 08 '22 07:09 ganduulgag

There is a command line option, --test-command that you can then provide a command to run the test. Something like --test-command "suricata -T" might work if using all the defaults.

If you are only using this tool for Suricata, you should really look at suricata-update which is bundled with Suricata these days. Its had some work to reduce memory usage over time, and runs suricata -T by default.

jasonish avatar Sep 08 '22 15:09 jasonish