opengrok

opengrok copied to clipboard

Is your feature request related to a problem? Please describe. As OpenGrok dynamically generate annotations by git-blame, when request a large file (20K+ lines) with annotations, it takes a long time (~2 mins) to receive the response.

Describe the solution you'd like Can you generate and cache the annotations during indexing? Just like history caches.

Describe alternatives you've considered None

Additional context None

I have a work-in-progress development branch that was supposed to introduce annotation caching, albeit only for the non-initial annotation view requests. It might be worthwhile to take the performance hit and generate the annotation cache (for the latest known revision of each eligible file) during indexing. This will definitely be non-default as generating annotations is usually very time consuming. Displaying older annotations or for out-of-sync files would still trigger the flow path which I guess is okay.

For the record, initial testing of my changes (where annotation cache is created for each file whose index document is updated/created) with the classic XML encoder/decoder shows significant increase of the indexing time. For the Lucene Git repository, it takes 1 minute to index it with standard settings (history cache enabled) and 13 minutes to index with annotation cache enabled. This is definitely CPU bound process - in the 2nd half of the indexing, where the annotation cache gets created, the CPU was completely used. So far it seems that the majority of that extra time is spent in the JGit methods, trying to retrieve the annotation, however the default Java XML serialization is definitely subpar performance wise, esp. in threading environment. As for disk space, the impact is minor - comparable to history cache size. So, just to be aware what you are asking :-)