datafusion-ballista icon indicating copy to clipboard operation
datafusion-ballista copied to clipboard

Ballista Executor report plan/operators metrics to Ballista Scheduler

Open mingmwang opened this issue 3 years ago • 1 comments

Which issue does this PR close?

Partial closes #116. There are still some pending changes need to be made to the DataFusion code base. Basically this PR covered those parts:

  1. Enhance rpc protocols to report task metrics rpc UpdateTaskStatus (UpdateTaskStatusParams) returns (UpdateTaskStatusResult) {}
  2. When the task finish, collect the plan metrics and report back to Scheduler
  3. In Scheduler side, merge and aggregate task metrics to stage metrics, update the ExecutionGraph.

Rationale for this change

What changes are included in this PR?

Are there any user-facing changes?

mingmwang avatar Aug 10 '22 11:08 mingmwang

@thinkharderdev @andygrove

Please help to review my PR.

mingmwang avatar Aug 10 '22 11:08 mingmwang

is there any other comments or concerns to be resolved before merging this PR? I haven't been following ballista closely so I don't know who has been merging the PRs recently

alamb avatar Aug 15 '22 16:08 alamb

@andygrove @thinkharderdev Can we merge this PR first? We can have ongoing PR to track the proper place for metrics and task status storage.

mingmwang avatar Aug 16 '22 09:08 mingmwang