cea-sec
Redis database backend
This PR introduces a new database backend: Redis.
Redis was designed to be an in-memory data store (cache), not a fully featured relational database, but it also implements 2 types of persistence: RDB, which is more like a periodic snapshot, and AOF, which gives real durability similar to what real databases can provide.
Since the data OpenWEC stores in its database is not particularly huge and the structure of the data is relatively simple (no complex table joins, etc.), Redis can be a good candidate for very fast and scalable data storage. We didn't measure the performance of OpenWEC with CockroachDB, so referring to performance is not completely fair, and I am sure that with proper configuration OpenWEC would reach the same scaling possibility and speed with PostgreSQL as with persistent Redis, but storing volatile data like Heartbeats/stats and frequently changing unstructured data like Bookmarks in Redis seemed an intuitive choice.
Our main "trigger" for this PR however was the simple fact that other commercial implementations of WEC rely on Redis as a Bookmark backend. Supporting Redis would make the transition to OpenWEC much easier for those who already operate their Redis clusters
This implementation is not yet complete, though it is functional, aside from the migration aspect. Consider this a first step - a foundation for further discussion. Redis operates quite differently from relational databases, which adds complexity to the implementation. I deliberately avoided altering the original trait definition, even though some adjustments for uniformity could have simplified the process.
The core idea is to store each item in Redis using a key structure like {type}/{subscription}/{machine}, ensuring unique entries. For example: HEARTBEAT:1231234124:my-computer. The associated data types can vary based on the specific information that needs to be stored.
To address the absence of SQL-like query features, these have been implemented on the client side (e.g., for stats). The implementation of the migration part might require further discussion. While structural changes in Redis are unnecessary, updating existing data across versions could be crucial.
Thanks for this huge pull request! :+1:
I will take a look at it in the next few days.
Just a first quick note: You don't need to "translate" existing migrations. A migration needs to be created when we make a change that alters the database schema or its content. Its goal is to ensure that OpenWEC users can safely upgrade from one version to another without having to reset their database (which is a must for production use).
For a new backend, you may want to create an initial migration to initialize the db schema if the DBMS requires it. If not, just make sure that we will be able to write migrations in the future to alter the db content.
Thank you for the review and helpful comments! I’ll need a couple of days to apply the suggested changes.
I believe I’ve addressed all of your requests, though all the request-change commits are prefixed with fu, which stands for FIXUP (not what you might think 😅). I can rebase these into a single request changes commit, or even merge them into the existing commits if you prefer. If you'd like to review them in smaller parts, feel free to check them in their current form. I hope the changes are good, but please let me know if you have any further suggestions or requests.
Wow, that's a lot of work! Thank you so much!
I'll do my best to review it in the next few days.
It was actually my pleasure! I’ve learned a lot of new things about Rust again. :)
Hi vruello, I hope you're doing well! Just wanted to kindly check in on this PR. I'm happy to make any further changes if needed, please let me know how I can help move this forward. Thanks for your time!
Hi! Thanks for the reminder, I did not forget this PR! :) I have been away from work for a few weeks, but I'm back now and I will review this as soon as I can :-)
Are you already using the Redis backend anywhere?
Are you already using the Redis backend anywhere?
No, we are currently testing it only locally, there are no real deployments yet.
