restatedev
Cron like scheduling primitives
It might be useful to be able to periodically invoke a function. The way such a cron function would be invoked is by an interval that is configured by the user.
Related to https://github.com/restatedev/restate/issues/844 as well.
Current proposal of the semantics for a first version:
- Schedules are expressed using the Cron format with second precision. This is an extension of the cron format used by
crontab, used in a couple of projects such as Spring and Quartz. - The cron is configured using an annotation on the handler, e.g. in Java/Kotlin
@Scheduled(cron = "10 * * * * *"). The registration of crons is static and associated with the event of registering/updating a service. - Only regular service handlers can be annotated with the
@Scheduledannotation. Virtual objects and Workflows handlers cannot use it (for now). - Through the CLI, a bunch of commands are exposed to observe the list of existing crons, the last X execution failures (if any).
- The default behaviour on missed ticks is to skip the invocation when the endpoint is down. In other words:
- If the endpoint is down, or takes long to reply, and a tick passes, that tick is missed.
- If the runtime is down, and at least one tick passes, the last tick should be executed. In practice, on recovery, Restate needs to figure out whether at least one tick elapsed, and needs to execute this.
- The handler MUST be empty request/empty response.
- On re-discovery, old crons are removed and new ones are registered. The first execution will be the next tick. In-flight invocations will continue to be processed by the old endpoint, as usual.
An extension of this proposal is to have an Admin API/CLI command or perhaps even Context API where crons can be dynamically created/removed, but perhaps we can do that once we get user feedback that this is effectively needed.
For posterity: Some implementation ideas can be found here.
This would be extremely useful.
The use case where we emulated scheduled services by implementing a recursive delayed object_send in a VirtualObject was pretty much as follows:
- Get the last processed state from context
- Check the state of an external system
- If the two are different, trigger a method to process the new state
- Schedule another check by making a call to
selfwith a 10 second delay
Slight difference here would be that the service would make a call to a VirtualObject holding the last processed state, but overall the flow could be easily implemented without the recursion hack.
Big fan of that feature! It would essentially enable Restate to sense (by polling) external systems that might not have the ability to push.
We are moving our BPMN execution engine over to restate. BPMN start events can be timer type start events and are configured using the ISO8601 repeating interval standard. https://en.wikipedia.org/wiki/ISO_8601#Repeating_intervals
Each time the timer triggers, it will call a service which will begin the execution of the BPMN process
My use case would be a static cron, not dynamic on application events, to start a business process at the same time on the same weekday each week, e.g. 12:00 Monday
I will (future) need to be able to start the same business process at different times with different inputs, e.g. 12:00 Monday with ABC input and 13:00 Monday with DEF input. I see inputs are out of scope for current proposal, of course it's possible to define a dedicated handler for each to invoke the main handler with the desired input in the meantime. This in my case won't need to scale up to loads of different inputs
I would also like to be able to "stop" the business process from running some days on demand, and turn it back on again, dynamic to application events, but for this I'd probably want to just have a virtual object with some boolean state that acts as that decision gate, which the cron-invoked handler would first check before continuing. I would still expect the cron to receive a success response from the handler so it doesn't retry. Again, I don't believe anything in this paragraph needs anything specific built to support it, just context for you
Something worth considering: Are cron expressions going to be UTC, or will it be possible to specify a timezone location, e.g. "London" that auto switches between UTC+0 (GMT) and UTC+1 (BST)
Just ticket and associated discord thread. +1 scheduling based on time/date vs delay duration. Is this still something that's being considered?
As an example use case: I've got a process that syncs updates new data from an external service on a regular schedule. If I rely on naive delays my process execution will drift over time, or I'll need to adjust the delay each time I set it which leads to other other complexities.
@kpwebb this feature is still on our radar and we plan to work on it soon, we just got sidetracked by other issues :)
Hi there, any updates on this? This would be game changing.
This feature is still very high on the list of new features that we want to add to the system in the next versions. Unfortunately, we didn't manage to complete it with the latest 1.3.0 release.
Hi there, any updates on this? This would be game changing.
This feature is still very high on the list of new features that we want to add to the system in the next versions. Unfortunately, we didn't manage to complete it with the latest 1.3.0 release.
Excellent! Thank you so much for all you have done!
Adding one comment I'm not sure if is captured yet. I'd love for it to be able to do timezone aware cron. Being in the US, a lot of crons need to be able to adjust on daylights savings.
Just want to add that our usecase for cronjobs is dynamically created ones, would appreciate if there is a way to make use of the static ones if possible, potentially with some config passed through
Not quite cron-style, but our team has been using this approach for periodic scheduling (wraps Restate client SDK), in case anyone finds it helpful! https://gist.github.com/brainhart/38599ea22db069f1aca4e20a07cd21b4
It's designed to run an idempotent function every period_ms, here's an example of how it's used:
export const periodicTask_cleanupTable = makeSingletonPeriodicTask({
name: "periodic_cleanupTable",
period_ms: 1000 * 60 * 60 * 12, // 12 hours
func: async () => {
await cleanupTable()
},
})
Hi! Is there any update on this? I'm set on using Restate to replace Airflow, and a Cron-like Scheduler would be the final missing piece
