json-log-viewer
json-log-viewer copied to clipboard
hedhyw
It froze our production server while reading a 288MB log file
I installed this log viewer on our production server after switching to pino for logging.
It took a moment to to load but it eventually loaded. I checked a couple of entries and pressed up to reload the file.
I was waiting for the file to load but I got a notification from our monitoring service that the production api server was down. I logged in to AWS to reboot the server from their EC2 dashboard. We were back online after 12 min.
300 MB is not huge for a log file. Even if it cannot handle big files, it might be good to have some check/limits and crash with a message rather than clog the whole system.
Ohhh, I'm sorry to hear. I hope this didn't have any big consequences.
I will add some reasonable limit until stream reading is done: https://github.com/hedhyw/json-log-viewer/issues/55
Big memory & CPU optimization included in https://github.com/hedhyw/json-log-viewer/releases/tag/v0.4.0
What changed:
- We still load the whole file (because we need to filter), but JSON parsed lazily (100 rows from config).
- Threshold between reloads (1 second from config).
- Reloading indicator.
- Limited the maximum size of the file (1 Gb from config).
- I've tested it on 600Mb file. It flies, but changes are significant, so we need to use more.
Please confirm that v0.4.0 works for you as expected!
I tried with version v0.4.0 and it still froze the whole server. I am using a staging server this time :) I am reloading the file (hitting up multiple times) and sometimes might trigger the reloading before it is done? Attached is a ~400MB redacted log file for you to replicate. redacted.example.zip
PS: is there any help or version command?
Good idea. I will add a version command.
Did you see "reloading..." when you pressed up? Was level "INFO"?
Was loading fast? Please, check the video: Screencast from 01-06-24 21:05:28.webm