Problem: outgrowing cqengine

[yrashk]

In certain aspects, we are starting to outgrow cqengine, the indexing engine behind es4j and one day we'll likely need our own indexing engine. cqengine was a great way to bootstrap and still is a good way to do what we want, but the time to upgrade will likely come. The telling signs so far are, in no particular order:

• cqengine doesn't have support for atomic multi-index updates (this breaks index consistency, albeit for a relatively short [depending on requirements] period of time); or transactional multi-index updates
• cqengine leaks details of its API that add a lot of noise to es4j apps. Consider Query<EntityHandle<..>> or ResultSet<EntityHandle<..>> versus Query<...> or ResultSet<...>
• cqengine can return only primary collection's data in a join query getIndexedCollection(Car.class).retrieve(query) returns ResultSet<Car>
• using cqengine API makes it harder to standardize our own index definition and query language standards across languages.
• the API design of it relies on generic type information and this limits our abilities to an extent. we rely on reflected Fields to extract that information, cqengine itself does the same through typetools (which uses undocumented sun.reflect API); not sure if we can improve this significantly within Java's constraints (can we do this in Java 9?), but worth look at, at the very least.
• limited expressibility of ordering-based queries ("find latest", etc.)

Proposed solution: implement our own indexing engine.

We can certainly learn from the experience with cqengine and take it to the next level. This issue is an open-ended "proposal collector" to figure out a longer term plan for a potential transition, or any ideas for any other way to address the problem.

(It is, of course, a significant undertaking)