tstack
Filter out old messages before load
Is your feature request related to a problem? Please describe.
Often it is convenient to load a large amount of archived/rotated log files collected in a folder, even though the time range of interest is only present in some of the files. These files may have been collected from a lot of different devices, may have different naming standards, archive formats etc.
Describe the solution you'd like Make it possible to define the times of interest before loading files. Lnav should then analyze the messages when loading, and discard messages not matching. Files containing messages of interest should be closed.
Describe alternatives you've considered Globbing files on load. Can be complex, doesn't work with individual messages.
Filter-lines-before will not close files or avoid the issue with many open files.
Additional context Lnav has issues loading a large amount of files (see #1115).
(for reference, #1188 has some conversations about this as well)
Thanks for the detailed writeup. It's helpful to understand the use case.
Enough folks seem interested in this that I should probably do something. I'll think on it.
I've pushed some changes that try to address this. If you have some time, I would appreciate you giving the top-of-tree a try. (Note that I am still testing and working through issues.)
I've added the -S/--since and -U/--until command-line flags. These take a relative or absolute time and set the cutoff for files to be considered. If the contents of the file fall outside of the cutoff, it will be closed out. Otherwise, the whole file will still be shown (I still need to make the changes to limit indexing to a subsection of the file).
lnav will still index the first part of the file. But, once that is done, it will jump to the end to try and find the last line to determine time range of the file. Note that compressed files will need to be decompressed in order to get to the end. So, there can still be a lot of work that needs to be done in order to eliminate a file.
Seems fine for my typical use case, but I will need to run this on some test system or set of logs to find out if the performance is adequate. I don't have such a set of thousands of files handy, will need to check when at work, on a live system or preferably a long-running test setup.
Still doesn't solve the issue with parsing very many files, lnav crashes when opening 5814 files from two months, trying to use -S and -U to filter in the last two week. Using a set of 2280 files from only november, lnav works as expected.
Though i would like a date like "today" or "2025-11-10" to mean 00:00:00 if used "since", and 24:00:00 if used "until", so all of that day's logs are included
Sorry, no, not from that environment. Customer stuff. I don't have any other environment handy where I've collected that many files.
Would you be able to look through the debug log yourself and give me a hint as to what might be happening?
lnav crashes when opening 5814 files from two months
I've tried throwing thousands of files at lnav and I can't make it crash. It takes a long time to index and the UI isn't always showing progress, but no crashes. So, if you can get me some information about the crash, I would really appreciate it.
The logs are created by logrotate, gz compressed into daily archives for each subscriber. So, in an environment with a hundred subscribers there are about 3000 log files per month, gzipped, ranging from 0 kb to several MB/day