agronholm
Shared job stores
The top requested feature for APScheduler is the ability to have multiple schedulers running against the same job store. This requires complex coordination between the schedulers. I can think of at least two good candidates for a synchronization backend:
- Apache ZooKeeper
- HashiCorp Consul
Any backend for this needs to be capable of providing at least 1) distributed locks, 2) notifications when the scheduler must wake up. This rules out pretty much all "trivial" backends, like file-based locking.
This feature would be very interesting. I did not took the time to dig into the code, but when the scheduler is DB backed, a simple lock for reading in the DB should do the trick.
No it won't. How would the scheduler know to wake up and adjust its next wakeup time? If it doesn't, it could sleep past the next fire time.
Therefore a notification (pub/sub) mechanism is also required in addition to locking.
Hum, my bad, I should have looked more precisely. I thought jobs were checked periodically... like at a fixed period of time.
That would either severely affect the accuracy of the schedules (with a high interval) or cause a significant load increase (with a low interval). The reactive approach is the only reasonable solution.
Agreed. I'll just post here this link in which you discussed it a long time ago, because in my case, I don't need a to update my list of jobs, so the mongolock solution would work. https://groups.google.com/forum/#!topic/apscheduler/Gjc_JQMPePc It might also help people in my case.
I could see this working with PostgreSQL (with listen/notify) and MongoDB and probably Redis too, but Zookeeper, etcd and Consul would likely make the best backends for shared job stores because they were explicitly designed for things like this.
Is there any activities for this task? Or in the nearest future? I would like to be a part of it.
You could propose a high level solution. This should minimize the wasted effort in the event that I end up rejecting the proposal. Of course you can also write proof-of-concept code if you like.
this could be a very useful feature to build a redundant scheduling system
I've been writing proof-of-concept code in the last few days. Job store sharing and high availability will be the featureof APScheduler 4 that will stand out the most.
That is fantastic news. Thank you @agronholm. Sorry to be a pest but any ETA on release date (even RC)?
I think I gave someone an ETA last summer but I've way overshot that due to a multitude of factors, the most important of which is my available free time which has to be distributed over my many F/OSS projects. I suggest that you keep watching for commits against master to get a clue about the progress. Sometimes weeks or even months can go by without any progress on a particular project.
Hi @agronholm, I would also need this feature. I was checking the commits and found it was implemented in commit: https://github.com/agronholm/apscheduler/commit/f4e8c3a0242b082fa1ca6ed5c78094f8de5ba439. When can we expect new release ?
Somebody asked me around this time last year and I told them it's a few months away. I've since been sidetracked by other projects. My intention is to solve a few critical design problems this months and to implement task accounting (how many instances of a task are running on workers). I too want to get this over with since people ask me about apscheduler on a weekly basis.
I suggest that you subscribe to #465 for progress updates.