activemq
activemq copied to clipboard
apache
Added activemq-redis-store
LevelDB is deprecated and has been removed from main branch. So, not sure it makes sense to include on Apache upstream. I can merge on 5.16.x but it won't be maintained (as LevelDB) from 5.17.x.
@jbonofre I am not sure I understand. This is a Redis storage layer, not LevelDB. Are you saying that Redis is of no interest as a storage layer? We have a requirement for scheduled messages to be replicated across regions for DR purposes. KahaDB doesn't seem to support this. Redis has replication, thus I implemented the persistence. If this is of no interest, I'll just build a separate internal artifact and this does not need to become a contribution. Thanks.
@maxfortun the PR makes references to the now removed leveldb. Can you rebase and squash?
To be clear-- this provides a job store only. KahaDB (or other persistence store) would be used for the other stores? messages, tmp space, etc.
I commented in Jira that I doesn't make sense to me to merge at Apache. Contributor should keep on his own.
@mattrpav , you are correct, the contribution is just a scheduled message store that does not change any default behavior and needs to be specifically configured within a broker block if it is needed. I was trying to submit a PR against a 5.16.x branch. Should I rebase against 5.16.x or main?
Should probably need a dev vote. Adding such a store requires commitment from pmc and committers to maintain
Yeah I am a -1 on such a big change like this (adding an entire new store) without a discussion first. I find it highly unlikely this will be maintained properly based on the LevelDB experience so I'm skeptical about taking this.
Well, this is an optional add-only change. As it does not modify any of the existing behaviors, and as such is not compromising any of the existing functionality. So I would not classify this as a big change. From what I could find KahaDB doesn't seem to support cross-region replication. Is there another way to persist scheduled messages across brokers via a data store or a network? I am currently deploying this as a separate artifact in my AMQ runtime, and totally understand if you find that it does not benefit the ActiveMQ community.
I think this is a super interesting feature even if it doesn't end up getting merged. Thanks for submitting this @maxfortun!
I don't see a space for "community plugins" on the ActiveMQ website but I think it would be great to start compiling a list of unofficial plugins similar to Rabbit MQ for example: https://www.rabbitmq.com/community-plugins.html.
Not sure about this one as it's based on leveldb which is removed now. I would rather focus on redis and bookkeeper store similar to kahadb.
Not sure about this one as it's based on leveldb which is removed now. I would rather focus on redis and bookkeeper store similar to kahadb.
@jbonofre, I think the references to leveldb are just because of the branch or commit this was based on which still included leveldb. There doesn't seem to be any usage of leveldb at all in this redis store implementation.
Correct but I wasn't clear: it seems to be just a job scheduler store, not a message store. So for me it makes sense to be renamed as it's not a store but "just" a job scheduler store
Correct but I wasn't clear: it seems to be just a job scheduler store, not a message store. So for me it makes sense to be renamed as it's not a store but "just" a job scheduler store
Agreed.
Clarification:
The code was copied and augmented with Redisson client from InMemoryJobScheduler (https://github.com/apache/activemq/tree/main/activemq-broker/src/main/java/org/apache/activemq/broker/scheduler/memory)
The dir/file structure/layout were modeled after leveldb, as I thought that's how optional components are packaged into activemq.
Unlike leveldb, Redis is a standalone service, thus all processing is happening outside of activemq process space. It's closer to JDBC than to leveldb.
If you believe that Job Scheduler Store and Persistence Adaptors should be packaged differently, I can rename/change the dir structure.
Ultimately, if there was interest and traction by activemq maintainers, the intent was to reach out to redis people and see if they want to contribute.
On Mar 17, 2022, at 4:16 AM, Lucas Tétreault @.***> wrote:
Correct but I wasn't clear: it seems to be just a job scheduler store, not a message store. So for me it makes sense to be renamed as it's not a store but "just" a job scheduler store
Agreed.
— Reply to this email directly, view it on GitHub https://github.com/apache/activemq/pull/751#issuecomment-1070521974, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACNVZ5DB4SW3NIPPTO4JBI3VALS6JANCNFSM5M36KMRA. You are receiving this because you were mentioned.
I think as long as we have a good enough clean api for abstraction, additional stores and plugins can live else where outside the core activemq project. Now if something is not exposed via api/extension then of course enhancement to expose that to make things pluggable.
My understanding is that there is no need to add a storage function to activemq's Redis, as it can do cross machine clustering for Redis and store Redis data on disk. activemq only does professional work, thank you.
My understanding is that there is no need to add a storage function to activemq's Redis, as it can do cross machine clustering for Redis and store Redis data on disk. activemq only does professional work, thank you.
That works only for HA, there is no DR. If you want data persistence across active/standby clusters, currently there is no way to achieve it with activemq the way it is. At least I could not find how.
The activemq lock grabbing function enables him to achieve HA and persistence of data together with the database. My project environment has been deployed in this way for more than 10 years @maxfortun
The activemq lock grabbing function enables him to achieve HA and persistence of data together with the database. My project environment has been deployed in this way for more than 10 years @maxfortun
Have you tried deploying your architecture in multi-cluster/multi-region environments with no shared file system?
Implemented out of project https://github.com/maxfortun/activemq-scheduler-store-redis