[FEATURE]: Support running JOIN tasks synchronously

[lbestatlas]

Describe the Feature Request

We have recently tried running JOIN tasks synchronously in Conductor. To do this we made a small change to WorkflowExecutorOps.isLazyEvaluateWorkflow and updated the JOIN task to return isAsync = false.

Our reason for switching to sync are

1. predictable completion of the JOIN without the need to tune any settings
2. reduction in JOIN executions for long running JOINs

Describe Preferred Solution

Make existing JOIN task sync Pros

• simple change
• likely will be more performant for the majority Cons
• maybe some use cases it results in more JOIN executions

Describe Alternatives

1. Have the JOIN task be driven by a system property Pros

• this is an fairly simple change
• gives consumers the option, although still a single setting for all JOINs Cons
• may make maintenance of execution. logic more complicated

2. Have the JOIN task be either async or sync based on TaskDef

• would require isAsync to be passed the task which won’t work initializing queues
• might need to introduce a new method supportsAsync() Pros
• this is a flexible solution allowing workflow owners to choose Cons
• more complexity in making this change

3. Have both a sync and non sync JOIN task

• any conditional logic related to execution of JOINS will not be able to use the TaskType there will need to be another attribute of tasks i.e. WorkerSystemTask.isJoin() Pros
• this is a flexible solution allowing workflow owners to choose which JOIN they use Cons
• more complexity in making this change

[svelango]

We also have the same issue. Can you pls share the change you made on isLazyEvaluateWorkflow