Recoverable recurring task

[piotrooo]

trafficstars

Let's image the following task:

task_name	task_instance	task_data	execution_time	picked	picked_by	last_success	last_failure	consecutive_failures	last_heartbeat	version	priority
mailboxes-scheduler	recurring	null	2025-05-21 09:16:55.799585 +00:00	false	null	2025-05-21 09:16:25.799585 +00:00	null	0	null	3	null

This task should be persistent and protected from deletion. If someone deletes it for any reason, it should be automatically recreated (just like it is on application startup).

Is there a way to handle this?

The ideal solution would be to add an isPersistent() method to the RecurringTaskBuilder, along with a watchdog mechanism that monitors the state of recurring tasks. If a persistent task is missing, the watchdog should recreate it.

Does this make sense to you, @kagkarlsson?

Currently, I have something that relies on Kubernetes liveness probes, which restart the application if the scheduler is not running:

@Bean
HealthIndicator jobScheduler(JdbcTemplate jdbcTemplate, ApplicationContext applicationContext) {
    return () -> {
        boolean schedulerExists = Boolean.TRUE.equals(jdbcTemplate.queryForObject("SELECT count(*) > 0 FROM scheduled_tasks", boolean.class));
        if (schedulerExists) {
            AvailabilityChangeEvent.publish(applicationContext, LivenessState.CORRECT);
            return Health.up().withDetail("scheduler", "running").build();
        }
        AvailabilityChangeEvent.publish(applicationContext, LivenessState.BROKEN);
        return Health.down().withDetail("scheduler", "not-running").build();
    };
}