Does it load all file in memory?

[mvoidex]

We have logs with ~40kk lines

---

Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

[tstack]

No, it will only buffer a small amount of the file content in memory and keep a small index entry for each line. lnav does need to read through the whole file on start to build the index.

[mvoidex]

Ok, thanks.

lnav does need to read through the whole file on start to build the index

I see. It works fast for about 300k lines, but for 40kk there are lags. It'll be great, if index can be created in background. I understand, that it may be hard to implement, so you can close this issue.

[tstack]

Pardon my ignorance, but I don't know what you mean by "40kk"... "kk" is not a unit I'm familiar with.

If you have some time, can you post some performance numbers for the file you're loading. For comparison, on my i7 MacBook Pro, a 535MB web log with 3.5 million lines takes about 4 seconds to index.

[mvoidex]

40 millions, I'll post some more info (size and time to load) tomorrow

[mvoidex]

Size of file: 2868042209 Lines: 34107116

It takes 30 seconds to load and then it works laggy

(this log is also being updated at the time i open it)

[f9z]

Is it possible to display end of the file and then continue indexing? When large log is loading (10Gb+) it takes a while to get to the bottom i.e. last line