runtime icon indicating copy to clipboard operation
runtime copied to clipboard

[Scheduler]Cron seems inactive after a while

Open Hooghof opened this issue 1 year ago • 2 comments

On the insights instance, after running well for about 14 days, the flows starting with a scheduler and set to cron selector no longer appeared to start at the set time. Restarting the flows did not appear to solve the problem. It seems that the problem lies at a higher level, quartz perhaps? It turned out that the same flow in the test backend was still working, but not in production anymore.

After a short discussion with Raymond I found this, perhaps related? https://github.com/Talend/apache-camel/blob/master/components/camel-quartz/src/main/docs/quartz-component.adoc

Hooghof avatar Jun 12 '24 06:06 Hooghof

The link is the same as the official documentation:

https://camel.apache.org/components/next/quartz-component.html

Based on the documentaiton it may happen because of a "misfire". A "misfire" means they are not executed as expected at the scheduled time. A misfire can have many cause, but in our case I think it's either running out of resources (CPU/Memory) or a concurrency issue. Other less likely causes are time syncrhonization problem, misconfiguration or network latency.

The idea is to have the following steps:

  1. Check if this also happens in Camel 4.
  2. If so change to configuration (in the frontend):

Add interruptJobsOnShutdown=true Add recoverableJob=true Add &trigger.misfireInstruction=2 (of 3 of 4)

  1. If still causes problem than it also possible to create a custom scheduler so we may add more logging to it. By default this should be avoided.

As alternative we may test if we can use the cron component as this doesn't use quartz. For the repeater we can use the timer (or scheduler) component.

skin27 avatar Jun 14 '24 09:06 skin27

Move this to active. IF we run into same problem then reopen this issue with all remarks.

skin27 avatar Jun 24 '24 07:06 skin27