stamparm
"Slow Loading" Why maltrail uses only one core?
Hi,
I realized something. I have a big "dprk.txt" in /trails/custom/ directory. It's almost 20mb. When I started maltrail, the process takes very long time. I know it's reading from file and writing to the buffer but; why is it using only one core? I have many cores:)
Did you think that? @stamparm
Loading of custom TXT should not take too long. Updates are way slower. Also, loading of trails is something that is being done once (on load) and that's it. Is there a way for me to reproduce your experienced behavior? Also, how long does it take in your case?
@stamparm
For example, I'm updating maybe 5-6 six times my custom trail text. In my case, approximately 15 minutes it takes. 5x15 = 75 minutes lost. It is not a problem that maltrail fetching from threats from sources. The main problem is when maltrail starts to loading /root/.maltrail/trails.csv to the buffer. It just uses only one core. Then it reproduces.
Just try it, use a big custom text file in your custom trail directory.
Can you please explain this part (2x2 6gb)?
What's the size of that custom trail file(s)? You've mentioned in previous message 20mb and now I see gigabytes.
p.s. just a friendly tip. If you are expanding IP ranges and using those as trails, from couple of months you are able to use condensed IP ranges format (e.g. x.y.z.w/u or even x.y.z.w-a.b.c.d)
Still, just wonder that why it doesn't work multiprocess?
-because nobody ever loads configuration files (and/or trail files in this case) in multiprocessing mode. Multi-processing is used for high CPU intensity tasks, like math calculations, hash cracking, etc. Loading of textual files is something that should not be classified as CPU intensive task. It is more a stress to other resources, not CPU
I am just trying to understand what is going on in your case and trying to replicate the behavior
@stamparm I understand that. But if you want to understand what i mean; just start maltrail with 10 million IP adresses in a custom text file with diffrerent ram sizes. Wait forever.
Ok. That's why I've said If you are expanding IP ranges and using those as trails, from couple of months you are able to use condensed IP ranges format (e.g. x.y.z.w/u or even x.y.z.w-a.b.c.d)
Have you manually expanded those IPs from IP ranges?