kestra icon indicating copy to clipboard operation
kestra copied to clipboard

Chore/worker refacto

Open loicmathieu opened this issue 8 months ago • 1 comments

Currently we use 2 thread for each worker job:

  • One thread of the main executorService which is used immediately after queue consumption to not block the queue. It also limit concurrency of job execution.
  • One thread to effectively execute the task or evaluate the trigger, this is needed to be able to interrupt it.

This PR keeps the first thread pool as we don't want to block the queue reception (or we will deadlock in JDBC), and we need to limit concurrency. But it then uses a virtual thread to execute the task / evaluate the trigger to reduce resource consumption and switch from the low level thread API to a Callable<State.Type>.

loicmathieu avatar Jun 17 '24 08:06 loicmathieu