go-quartz
go-quartz copied to clipboard
reugn
Usage in clustered environment
How do you use this in a clustered environment where more than one instance of the scheduler might exist? Is there a feature similar to quartz cluster mode (https://flylib.com/books/en/2.65.1/how_clustering_works_in_quartz.html)?
@gitkeerthi Clustering is not supported within the library, but a persistent job store can be implemented independently.
Maybe a simple global system like etcd , kine or nats would be a simple global locking store .
let’s use NATS because it has subscriptions which is handy.
so the 3 events are:
When a job starts it registers with NATS , so no other servers take the job.
on job start , check with NATS and if not in nats, run the job.
On job finish , unregister with nats
—-
you can embed nats servers in your project too which is handy . Nats will automatically do leader election so if a server fails and new leader is automatically chosen . It also has gracefull upgrades using lamb duck mode. So this means you don’t have to run a “special jobs locking server” but can just include it in your golang project
—-
I don’t know if Quartz has job lease time and retries , but the above can probably work with the above design .
Btw thos already exists and uses nats for job locking as well as other aspects :
https://github.com/choria-io/asyncjobs
it also has a schedular: https://choria-io.github.io/asyncjobs/overview/scheduled-tasks/
in their code it’s used cron golang module.
maybe there is opportunity to use Quarz but not sure yet.
@gitkeerthi, @gedw99, @bingtianyiyan I've opened a pull request (#80) to support custom job queue implementation that can provide persistent storage and state sharing for jobs. Please review, and let me know if this satisfies your requirement.
You mean that we can create multiple instances of Scheduler, and these schedulers can have respective Persistent Queue(We can surely set a field in RDBMS table to seperate each queue jobs)? Each scheduler has own timer, event-loop etc. Please correct me if I misunderstand this solution.
By implementing a shared persistence layer, you can enable multiple scheduler instances to retrieve and process jobs. This architecture supports high availability (HA) scenarios and facilitates scalability when job processing becomes highly competitive.