incubator-gluten
incubator-gluten copied to clipboard
[VL] Metrics for Scan seems wrong
Backend
VL (Velox)
Bug description
I was running TPCH Q1, the details for the Scan is:
Scan parquet spark_catalog.tpch1g_parquet.lineitem
number of skipped splits: 0
totaltime of scan and filter total (min, med, max (stageId: taskId))
157 ms (77 ms, 80 ms, 80 ms (stage 0.0: task 0))
number of files read: 12
io wait time total (min, med, max (stageId: taskId))
12 ms (6 ms, 6 ms, 6 ms (stage 0.0: task 0))
totaltime of scan total (min, med, max (stageId: taskId))
0 ms (0 ms, 0 ms, 0 ms (stage 0.0: task 0))
number of output bytes total (min, med, max (stageId: taskId))
633.9 MiB (314.8 MiB, 319.2 MiB, 319.2 MiB (stage 0.0: task 0))
number of skipped row groups: 0
number of raw input bytes total (min, med, max (stageId: taskId))
0.0 B (0.0 B, 0.0 B, 0.0 B (stage 0.0: task 0))
cpu wall time count: 1,474
number of output vectors: 1,470
number of raw input rows: 6,001,215
metadata time total (min, med, max (stageId: taskId))
2 ms (0 ms, 0 ms, 2 ms (driver))
number of preloaded splits: 0
size of files read total (min, med, max (stageId: taskId))
196.0 MiB (0.0 B, 0.0 B, 196.0 MiB (driver))
peak memory bytes total (min, med, max (stageId: taskId))
10.7 MiB (5.4 MiB, 5.4 MiB, 5.4 MiB (stage 0.0: task 0))
number of processed row groups: 0
number of output rows: 5,850,871
number of memory allocations: 26,600
remaining filter time total (min, med, max (stageId: taskId))
0 ms (0 ms, 0 ms, 0 ms (stage 0.0: task 0))
number of processed splits: 0
number of dynamic filters accepted: 0
Note that the following two metrics are all zeros:
- totaltime of scan total
- number of raw input bytes total
Seems wrong?
Spark version
Spark-3.2.x
Spark configurations
No response
System information
Velox System Info v0.0.2 Commit: 17210d9e039815af7cfb824e90cdbade49a464e9 CMake Version: 3.29.0 System: Darwin-23.1.0 Arch: arm64 C++ Compiler: /Library/Developer/CommandLineTools/usr/bin/c++ C++ Compiler Version: 15.0.0.15000040 C Compiler: /Library/Developer/CommandLineTools/usr/bin/cc C Compiler Version: 15.0.0.15000040 CMake Prefix Path: /Library/Developer/CommandLineTools/SDKs/MacOSX14.0.sdk/usr;/opt/homebrew;/usr/local;/usr;/;/opt/homebrew/Cellar/cmake/3.29.0;/Users/abei/Code/velox_workspace/installed;/usr/X11R6;/usr/pkg;/opt;/sw;/opt/local
Relevant logs
No response