How to run a Camel process as a cron job (other than a timer)

[brunoNetId]

With the following use case as an example:

• process the daily Kafka messages produced in a topic

How could we run a Cron job with Camel K that connects to Kafka, consumes the latest messages, and shuts down when no more messages are available (e.g. setting a timeout).

I've created a proposed example (attached) where I use an aggregator with a timeout. When the aggregator completes, let's say when no new messages arrive in a specific window of time, a shutdown process trigger to stop the CamelContext and the Java process.

Cron could see the process has stopped and schedule the next run the next day or as per a Camel timer definition.

For simplicity, the example uses HTTP instead of Kafka. cron-example.tar.gz

[christophd]

there is a cron trait that may be useful for this: https://camel.apache.org/camel-k/1.9.x/traits/cron.html

[brunoNetId]

Yes... unfortunately, the trait seems limited and not able to cover the proposed use case.

[lburgazzoli]

The issue with the cron trait for this particular case is that the camel context is stopped after the first exchange completes, however if an aggregation policy is defined, then the cron would kill the pod before the aggregator completes.

We should probably enhance the shutdown strategy to handle this case

[davsclaus]

You can use the camel.main.durationXXX options where you can say that if Camel is idle after 30 seconds (then terminate). You can then use from kafka in the route, that would then drain the topic until there are no more messages. Then Camel becomes idle and after X period it terminates itself.

And use the cron trait to make it run once per day or what schedule you need.

[brunoNetId]

As per @lburgazzoli, the problem there is that all Camel exchanges may have terminated, but not the aggregator... Camel may think it is idle, but it is actually not, the aggregator is still ongoing.

Another problem is that currently the cron trait only works with timer and passive components... not if a kafka consumer is defined (I think)

[github-actions[bot]]

This issue has been automatically marked as stale due to 90 days of inactivity. It will be closed if no further activity occurs within 15 days. If you think that’s incorrect or the issue should never stale, please simply write any comment. Thanks for your contributions!