incubator-gluten icon indicating copy to clipboard operation
incubator-gluten copied to clipboard

[VL] Metrics for Scan seems wrong

Open xumingming opened this issue 1 year ago • 0 comments

Backend

VL (Velox)

Bug description

I was running TPCH Q1, the details for the Scan is:

Scan parquet spark_catalog.tpch1g_parquet.lineitem

number of skipped splits: 0
totaltime of scan and filter total (min, med, max (stageId: taskId))
157 ms (77 ms, 80 ms, 80 ms (stage 0.0: task 0))
number of files read: 12
io wait time total (min, med, max (stageId: taskId))
12 ms (6 ms, 6 ms, 6 ms (stage 0.0: task 0))
totaltime of scan total (min, med, max (stageId: taskId))
0 ms (0 ms, 0 ms, 0 ms (stage 0.0: task 0))
number of output bytes total (min, med, max (stageId: taskId))
633.9 MiB (314.8 MiB, 319.2 MiB, 319.2 MiB (stage 0.0: task 0))
number of skipped row groups: 0
number of raw input bytes total (min, med, max (stageId: taskId))
0.0 B (0.0 B, 0.0 B, 0.0 B (stage 0.0: task 0))
cpu wall time count: 1,474
number of output vectors: 1,470
number of raw input rows: 6,001,215
metadata time total (min, med, max (stageId: taskId))
2 ms (0 ms, 0 ms, 2 ms (driver))
number of preloaded splits: 0
size of files read total (min, med, max (stageId: taskId))
196.0 MiB (0.0 B, 0.0 B, 196.0 MiB (driver))
peak memory bytes total (min, med, max (stageId: taskId))
10.7 MiB (5.4 MiB, 5.4 MiB, 5.4 MiB (stage 0.0: task 0))
number of processed row groups: 0
number of output rows: 5,850,871
number of memory allocations: 26,600
remaining filter time total (min, med, max (stageId: taskId))
0 ms (0 ms, 0 ms, 0 ms (stage 0.0: task 0))
number of processed splits: 0
number of dynamic filters accepted: 0

Note that the following two metrics are all zeros:

  • totaltime of scan total
  • number of raw input bytes total

Seems wrong?

Spark version

Spark-3.2.x

Spark configurations

No response

System information

Velox System Info v0.0.2 Commit: 17210d9e039815af7cfb824e90cdbade49a464e9 CMake Version: 3.29.0 System: Darwin-23.1.0 Arch: arm64 C++ Compiler: /Library/Developer/CommandLineTools/usr/bin/c++ C++ Compiler Version: 15.0.0.15000040 C Compiler: /Library/Developer/CommandLineTools/usr/bin/cc C Compiler Version: 15.0.0.15000040 CMake Prefix Path: /Library/Developer/CommandLineTools/SDKs/MacOSX14.0.sdk/usr;/opt/homebrew;/usr/local;/usr;/;/opt/homebrew/Cellar/cmake/3.29.0;/Users/abei/Code/velox_workspace/installed;/usr/X11R6;/usr/pkg;/opt;/sw;/opt/local

Relevant logs

No response

xumingming avatar Mar 24 '24 13:03 xumingming