Race between Suggester and Indexer leaving Suggester DBs uninitialized

[aerofeev2k]

In this scenario I'm bootstapping everything, meaning that I'm not indexing project per project, I'm rehearsing a major version upgrade and am indexing the entire source directory from scratch with -P -S and no configuration present/loaded anywhere (partly because the format is not backward compatible).

In this -P -S mode OpenGrok is discovering all projects/repositories, then indexing everything project by project and is pushing indexed=true flag to the webapp as soon as a project is ready. It's a nice way to notify webapp about project readiness, but unfortunately this doesn't work as the webapp doesn't know anything about projects yet, because the entire config would be pushed to it only at the end of indexing of ALL projects.

So, not wanting to wait and and stare at an empty webpage, I ran OpenGrok twice. First with -P -S -n to generate config file with all the projects and repositories and to push it to the webapp; then only with -P -S, and this did the actual indexing, while properly pushing indexed=true to the webapp, which the webapp was properly accepting, now that it had the config preloaded.

This worked great, and the projects were showing up in the GUI as soon as they were indexed, exactly as I wanted it, but Suggester apparently didn't like it.

[09/Apr/2022:09:41:47 -0400] "PUT /source/api/v1/projects/PROJECT/indexed HTTP/1.1" 202 -
09-Apr-2022 09:41:47.614 INFO [ForkJoinPool.commonPool-worker-115] org.opengrok.suggest.Suggester.rebuild Rebuilding the following suggesters: [PROJECT]
09-Apr-2022 09:41:47.715 WARNING [ForkJoinPool-2-worker-51] org.opengrok.suggest.SuggesterProjectData.initFields Fields [defs, path, hist, refs, type, full] will be ignored because they were not found in index directory MMapDirectory@/opengrok/data/index/PROJECT lockFactory=org.apache.lucene.store.NativeFSLockFactory@18746886
09-Apr-2022 09:41:47.746 INFO [ForkJoinPool.commonPool-worker-115] org.opengrok.suggest.Suggester.shutdownAndAwaitTermination Suggesters for [PROJECT] were successfully rebuilt (took 0 seconds)

Webapp got properly notified of the project's readiness, the project showed up in the GUI, Suggester got kicked off, but it didn't really index anything and left only /opengrok/data/suggester/PROJECT/version.txt file with "1" in it. Restarting Tomcat didn't do much, and only blowing away /opengrok/data/suggester/PROJECT triggered regeneration, which this time completed successfully. I imagine, another way for me to fix this would be to make a change in the project source directory and reindex the project to bump up the main index version.

My thinking is that the indexed=true notification arrives to the webapp a little bit too early when the index isn't fully ready yet, and so Suggester can't open or find something in it. There is also an issue with leaving a version.txt in the Suggester's project directory, which prevents Suggester from rescanning the main index N hours later on its own and self-healing.