dolphinscheduler [Improvement-16979] Unify PriorityDelayQueue with AbstractDelayEventBus

Purpose of the pull request

implement #16979

Brief change log

Refactored DelayEntry.java,PriorityDelayQueue,PriorityAndDelayBasedTaskEntry,TimeBasedTaskExecutionRunnableComparableEntry with TaskDispatchEntryEvent,TaskDispatchEntryEventBus which inherited from AbstractDelayEventBus,AbstractDelayEvent
Fix PriorityAndDelayBasedTaskEntry's compare issue, old object compare task priority after compare time in millisecond, cause the task priority only affect in same millisecond, now it will compare task priority first, then the create time.

Verify this pull request

the test codes were refactored.

Pull Request Notice

If your pull request contains incompatible change, you should also add it to docs/docs/en/guide/upgrade/incompatible.md

Apr 30 '25 02:04 reele

Good job, after this PR merged, the master code will be clearer. Base on this, we can provide more EventBus to expose the workflow instance/task instance lifecycle event to exposed to outside.

Apr 30 '25 06:04 ruanwenjun

Can we remove GlobalTaskDispatchWaitingQueue and GlobalTaskDispatchWaitingQueueLooper, this queue is used to handle the delay time, but bring some extra problems, e.g. there are existing many thread to deal with the task lifecycle in master, we need to do extra concurrency control, e.g. one task is in dispatching, and then receive a kill event.

Since TaskDispatchLifecycleEvent is already an DelayEvent so we can directly set the delay time to it, then we can removed TimeBasedTaskExecutionRunnableComparableEntry and WorkerGroupTaskDispatcher. We directly use ITaskExecutorClient to dispatch task in dispatchEventAction, if dispatch failed then regenerate a TaskDispatchLifecycleEvent with retry delay time.

i'll have a try.

Apr 30 '25 06:04 reele

@reele Sorry, I miss something, we need to sort the tasks by priority under one worker group, so WorkerGroupTaskDispatcher cannot be removed, but GlobalTaskDispatchWaitingQueue and GlobalTaskDispatchWaitingQueueLooper can be removed.

Apr 30 '25 16:04 ruanwenjun

@reele Sorry, I miss something, we need to sort the tasks by priority under one worker group, so WorkerGroupTaskDispatcher cannot be removed, but GlobalTaskDispatchWaitingQueue and GlobalTaskDispatchWaitingQueueLooper can be removed.

OK.

May 06 '25 03:05 reele

@reele Sorry, I miss something, we need to sort the tasks by priority under one worker group, so WorkerGroupTaskDispatcher cannot be removed, but GlobalTaskDispatchWaitingQueue and GlobalTaskDispatchWaitingQueueLooper can be removed.

OK.

I'll figure out how to fix this prioritization problem, which is in fact very inaccurate, and it's not sorting globally.

May 06 '25 07:05 ruanwenjun

I have removed GlobalTaskDispatchWaitingQueue at #17180

May 13 '25 15:05 ruanwenjun