opengrok
opengrok copied to clipboard
oracle
History is hogging web app memory
When thinking about #3243, I came to realize that the web app suffers from similar problem: in general whenever a history is requested, the History object representing the complete history of given file/directory is loaded into memory:
https://github.com/oracle/opengrok/blob/610d9085dcb52fc16b3f50d16e64cc03a62b7ea1/opengrok-web/src/main/webapp/history.jsp#L81-L84
What's more, it is actually stored in the request:
https://github.com/oracle/opengrok/blob/610d9085dcb52fc16b3f50d16e64cc03a62b7ea1/opengrok-web/src/main/webapp/history.jsp#L99
and used for paging.
Even for smaller repositories such as the OpenGrok Git repository this creates significant memory pressure. Here's a graph representing heap stats from a web app where I displayed history for couple of directories (including the top level directory) for indexed OpenGrok Git repository:
The Eden space grew by bunch of gigabytes !
The way how the History object is stored in FileHistoryCache makes it hard to do something about this. Firstly, it is compressed, secondly it is XML serialized Java object (#3539). So it has to be read whole into memory and then dissected.
Once a scheme to iterate over history (for both repository method and history cache) without reading all of it in memory is in place, the HistoryReader used during indexing should be converted to this scheme as well.