spark icon indicating copy to clipboard operation
spark copied to clipboard
verified publisher icon apache

[SPARK-54539][SQL] TaskMetrics add spillTime metrics

Open AngersZhuuuu opened this issue 1 month ago • 3 comments

What changes were proposed in this pull request?

TaskMetrics add spillTime metrics for task execution.

Why are the changes needed?

If we have this metrics, we can directly know that longer jobs have longer spill times, and we can optimize the job by adjusting the parallelism parameter.

Does this PR introduce any user-facing change?

After this pr then we can add spill cost in each corresponding execution node's metrics, then user can know the cost.

How was this patch tested?

Existed UT

Was this patch authored or co-authored using generative AI tooling?

No

AngersZhuuuu avatar Nov 27 '25 10:11 AngersZhuuuu

@cloud-fan How about this metrics? If it's ok, later I will add this in all related node.

AngersZhuuuu avatar Dec 03 '25 10:12 AngersZhuuuu

do we have similar metrics for SQL plan nodes?

cloud-fan avatar Dec 04 '25 18:12 cloud-fan

do we have similar metrics for SQL plan nodes?

If it's OK， later I will add this metrics to below SQL plan nodes. 截屏2025-12-05 10 21 28

AngersZhuuuu avatar Dec 05 '25 02:12 AngersZhuuuu

ping @cloud-fan How about this?

AngersZhuuuu avatar Dec 19 '25 02:12 AngersZhuuuu