RITA/MongoDB processes use too much memory and eventually cause the process to be killed

[awaken337]

Just want to start by saying how awesome RITA is! Thanks for sharing it with the community.

I recently installed RITA on a RHEL 7 machine using the install script. Its a beefy machine with 48 cores and 32 Gigs of RAM. The machine is also only running RITA; Zeek is installed on a different server. I imported 24 hours of Zeek logs and processed them with RITA using the default configuration. It took about 18 hours, but, the process finished and I was able to examine the results using RITA. I then added a couple IP ranges to the RITA config file so I wont see traffic from our public IP ranges to our internal IP ranges. Now, when I try to import Zeek logs into RITA, it will process one or two blocks of the 17 for a long period of time and eventually either MongoDB or RITA use so much memory that the process ends up getting killed:

At first, it was always Mongo that was using too much memory and getting killed. When I was troubleshooting that, I started the Mongo shell and saw a couple error messages that I addressed based on guidance provided by Mongo. Im using a NUMA machine, so I disabled zone reclaim and changed ExecStart to ensure Mongod is started by numactl. Additionally, I disabled transparent huge pages to help with MongoDB performance.

Now, when I try to import 24 hours of Zeek logs, RITA is the program that appears to be using more and more memory until the process ends up getting killed:

grep rita /var/log/messages Mar 16 11:00:33 securityappdev1 kernel: [45355] 0 45355 6043339 5737143 11486 115480 0 rita Mar 16 11:00:33 securityappdev1 kernel: Out of memory: Kill process 45355 (rita) score 556 or sacrifice child Mar 16 11:00:33 securityappdev1 kernel: Killed process 45355 (rita), UID 0, total-vm:24173356kB, anon-rss:22948572kB, file-rss:0kB, shmem-rss:0kB

I understand the value of the data RITA can provide and really want to get this working so I can do a POC and demonstrate RITA's capabilities to our team. Would these same issues arise if I installed RITA on a non-NUMA machine running Ubuntu 18.04? Is there anything I can do on my end to address this issue?

Thanks for your help!

[ethack]

If you are importing all 24 hours at once I'd recommend doing it an hour at a time instead. There's less for RITA to chew on at one time that way.

• https://github.com/activecm/rita#rolling-datasets
• https://github.com/activecm/rita/blob/master/docs/Rolling%20Datasets.md

TL;DR Rolling datasets have a fixed size (in chunks) and automatically remove the oldest chunk when they are full. So if you run the rita import every hour it will pick up the newest hour of logs and import that into a new chunk. The whole dataset has 24 chunks by default which then means you'll always have the latest 24 hours in it.

[bekirk]

I am having a similar issue I want to load 24 hours of data into a data set at on time. It says it is parsing into 1 of 2 batches, is there a way for me to tell it to maybe do 5 or 6 batches instead?

I just recently upgraded to rita 4.8.0 and on a old hardware based system 256Gb of RAM it works great but on a VM that I am certain I am going to be force to move to soon as the old hardware is probably 10+ years old and is on its last leg. The VM I am limited to 32Gb of RAM.

I can get through about 40 of the 108 files (18 Gigs compressed zeek logs) so I was thinking if I can process 20 or so files at a time it should stay well under the memory limit.

I know in the old version of rita it would split large data sets into 5-6 batches and on the weekends it was only like 2 batches. Is there a way to tell it how to split it up into batches without doing the rolling datasets?