egeria-connector-ibm-information-server
egeria-connector-ibm-information-server copied to clipboard
odpi
Explore performance / scalability improvement of stub storage
Investigate use of an external component (like Redis) for the storage of the OMRS stub information, to both improve overall performance of the event mapper as well as (probably) increase its scalability.
Such an improvement could also be made optional, based on whether the repository connector is configured with a property to refer to such an external component or not. (If not, it can simply fall-back to the default of using the existing OpenIGC stub mechanism.)
Initial look at Redis seems promising: memory footprint appears negligible (single-digit MB in official Docker container at https://hub.docker.com/_/redis), performance is blazing fast, and there appear to be mature clients for various languages:
- https://github.com/redisson/redisson
- https://github.com/lettuce-io/lettuce-core
- https://github.com/xetorthio/jedis
- https://github.com/luin/ioredis
Downside to Redis is that it does not appear to be something that can be easily run in an embedded way; alternative may be to investigate H2, which we're already using for some of the UI demos.
For redis there have.been a few attempts. https://www.baeldung.com/spring-embedded-redis may help?
Indeed, I saw that, but still feels like it's geared towards more of a unit-test-specific scenario... But maybe that's still sufficient nonetheless.
I'm trying to keep in mind a broader vision here as well that such a "cache" might be a useful, more broadly configurable option to include in the core of Egeria for anything to make use of that needs to store state in an efficient and scalable way: whether that be repository connectors or other servers / daemons. (That way we don't end up with lots of one-off persistence mechanisms that are all managed differently using different tools, but could leverage the more general connector framework of Egeria itself to define which (configurable) "cache" to use for various scenarios: including whether embedded, external, persistent vs. non-persistent, etc.
In that sense, it probably makes sense to first come up with the interface(s) and configurable options, and then which connectors we build to particular back-ends just gives us a variety of options for backing those interface(s) (?)
Will leave this open, but as the event mapper (only piece that requires the stubs) is now considered experimental, this is unlikely to be implemented.