datafusion-ballista
datafusion-ballista copied to clipboard
Ballista Executor report plan/operators metrics to Ballista Scheduler
Which issue does this PR close?
Partial closes #116. There are still some pending changes need to be made to the DataFusion code base. Basically this PR covered those parts:
- Enhance rpc protocols to report task metrics
rpc UpdateTaskStatus (UpdateTaskStatusParams) returns (UpdateTaskStatusResult) {} - When the task finish, collect the plan metrics and report back to Scheduler
- In Scheduler side, merge and aggregate task metrics to stage metrics, update the ExecutionGraph.
Rationale for this change
What changes are included in this PR?
Are there any user-facing changes?
@thinkharderdev @andygrove
Please help to review my PR.
is there any other comments or concerns to be resolved before merging this PR? I haven't been following ballista closely so I don't know who has been merging the PRs recently
@andygrove @thinkharderdev Can we merge this PR first? We can have ongoing PR to track the proper place for metrics and task status storage.