mtools
mtools copied to clipboard
mplotqueries --type connchurn uses a lot of memory
mplotqueries --type connchurn eats a lot of memory when there are a lot of connections being created/ended. I had a ~1GB logfile with ~5.5 million connections (so ~11 million lines), and mplotqueries --type connchurn was using over a GB of RSS after reading barely a few % of the file.
One way to address this is to let the plot types decide themselves what they want to keep after presenting them with the log line. This would also save time, as they only look at each line once, rather than multiple times.
- mplotqueries goes through file line by line
- Each plot type's
accept_line()
is called once. In this method, the plot type can decide to store any kind of information it requires for further processing - The grouping happens internally per plot type, and not globally in mplotqueries.
The problem with this approach is that plot types may have to store the logline multiple times, i.e. multiple scatter plots will each store the entire data.