Virat Bansal comments

Results 8 comments of


                                            Virat Bansal

[SPARK-49038][SQL] Fix regression in Spark UI SQL operator metrics calculation to filter out invalid accumulator values correctly

> Do we have a test case to demonstrate the issue? AFAIK Spark filters out 0-value accumulators at the executor side. `SQLMetrics.stringValue` has different logic for different metric types to...

[SPARK-49038][SQL] Fix regression in Spark UI SQL operator metrics calculation to filter out invalid accumulator values correctly

Just to note, `SIZE`, `TIMING` and `NS_TIMING` metrics are [created](https://github.com/apache/spark/blob/9f22fa4d2acfcbc42d6d76a28778885cbdad733d/sql/core/src/main/scala/org/apache/spark/sql/execution/metric/SQLMetrics.scala#L155-L182) with initial value `-1`.

[SPARK-49038][SQL] Fix regression in Spark UI SQL operator metrics calculation to filter out invalid accumulator values correctly

> Do we have a test case to demonstrate the issue? Added the test @cloud-fan

[SPARK-49038][SQL] Fix regression in Spark UI SQL operator metrics calculation to filter out invalid accumulator values correctly

> Are you sure it may contain -1? You can run the tpc-ds benchmark q1 and check the SQL tab DAG views in driver hosted Spark UI. The minimum and...

[SPARK-49038][SQL] Fix regression in Spark UI SQL operator metrics calculation to filter out invalid accumulator values correctly

> Do you have a simple repro (end-to-end query) to trigger this bug? Can you please use the below reproducer? This is join between two tables that shuffles data. This...

[SPARK-49038][SQL] Fix regression in Spark UI SQL operator metrics calculation to filter out invalid accumulator values correctly

@cloud-fan waiting for your response to unblock the review - thanks!

[SPARK-49038][SQL] Fix regression in Spark UI SQL operator metrics calculation to filter out invalid accumulator values correctly

> Can you please use the below reproducer? This is join between two tables that shuffles data. This can be run in a spark-shell. @cloud-fan were you able to reproduce...

[SPARK-49038][SQL] Fix regression in Spark UI SQL operator metrics calculation to filter out invalid accumulator values correctly

Thanks for checking and confirming! Actually I was to able to track down that the bug started after #39311 only and hence I proposed this fix. But If there is...