lnav
lnav copied to clipboard
tstack
Command line flags to load logs since a timestamp
Is your feature request related to a problem? Please describe.
A lot of times I find myself loading huge log files all together, however, I am only interested in examing entries after a specific timestamp. There is no way right now tell lnav not to load all lines, and with all the extra unecessary lines it slows down lnav.
Describe the solution you'd like
Too extra command line options --after and --before which should accept a datetime argument that could be used to limit the lines that are read into lnav. The rest of lines should be skipped at loading time.
Describe alternatives you've considered
Nil
Additional context
Nil
There are :hide-lines-after and :hide-lines-before commands that you can pass from the command line with the -c option.
extra unecessary lines it slows down lnav.
Is the initial indexing slow or interactive use? The initial index can't really be avoided right now.
Initial indexing. I regularly deal with tarballs of multiple GBs, and sometimes lnav crashes/hangs during initial indexing.
Initial indexing. I regularly deal with tarballs of multiple GBs, and sometimes lnav crashes/hangs during initial indexing.
What version are you using? Can you send some of the crash logs?
If you have some time, can you try one of the artifacts attached to one of the nightly builds to see if that fixes anything:
https://github.com/tstack/lnav/actions/runs/6093182503
I do not have any crash logs right now. It does not happen everyday, but I will collect one next time.
I am using 0.11.2, will try a nightly build next time when it crashes.
Today it crashed again when loading a 8.6GB directory. I tried both 0.11.2 and nightly build, they both crashed.
Unfortunately neither version generated any crash logs. There is one crash log from May 29th, which I guess could be by an older version. I am putting it in a gist any way, and hopefully it has what you are looking for.
Today it crashed again when loading a 8.6GB directory. I tried both 0.11.2 and nightly build, they both crashed.
Sorry, that sucks. Thanks for trying out the nightly build.
I would like to try to figure out what the problem is and get it fixed. Since I can't seem to replicate it, I would need some help from you. One option is for you to send the files to [email protected] , but they probably have sensitive data. You can get a debug log from lnav by passing -d /path/to/lnav.log on the command-line and then sending that. You could try starting lnav and then attaching to it with gdb/lldb, then you might be able to get a backtrace when it crashes (but, you would probably need to build from source to get reasonable line numbers).
It is probably a memory issue.
The machine I use to load log files only had 8GB of memory, and after increasing it to 16GB, I have tried loading the directory twice, it crashes no more.
Still, it would be great if we could filter out some data at loading time and save some memory.
@tstack Unfortunately the flag :hide-lines-before doesn't work exactly as I expect it to. I also have a file ~5Gb and it takes quite long time to load. The option does not reduce load time. Moreover, the search also works throughout the entire file despite the option. Is it the only option to limit lines?
Moreover, the search also works throughout the entire file despite the option. Is it the only option to limit lines?
@graceman9, can you give some more details? Do you see log messages with timestamps before the time passed to :hide-lines-before? The search should only be happening for displayed lines.
@tstack My bad, probably I used this option incorrectly.
Tried again - works great! Thanks
lnav -c ":hide-lines-before 2024-07-25" var/log/php.log