compass icon indicating copy to clipboard operation
compass copied to clipboard

[BUG]: Full table scan scenario check condition

Open ketingli1 opened this issue 2 years ago • 8 comments

大表扫描场景里,判断SparkPlanInfo节点是否为扫描节点的逻辑为:node.getNodeName().startsWith("Scan")

但是实际的扫描节点的名称并不是Scan开头,逻辑是否有误? "children": [{ "children": [], "metadata": {}, "metrics": [{ "accumulatorId": 21, "metricType": "sum", "nodeName": "number of output rows" }], "nodeName": "HiveTableScan", "simpleString": "HiveTableScan [id#6L, plan_type#10], HiveTableRelation dev.data_skew_table_partitioned, org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, [id#6L, task_pod_name#7, project_id#8L, create_time#9, plan_type#10, deleted#11], [ds#12]" }],

ketingli1 avatar Sep 19 '23 08:09 ketingli1

大表扫描场景里,判断SparkPlanInfo节点是否为扫描节点的逻辑为:node.getNodeName().startsWith("Scan")

但是实际的扫描节点的名称并不是Scan开头,逻辑是否有误? "children": [{ "children": [], "metadata": {}, "metrics": [{ "accumulatorId": 21, "metricType": "sum", "nodeName": "number of output rows" }], "nodeName": "HiveTableScan", "simpleString": "HiveTableScan [id#6L, plan_type#10], HiveTableRelation dev.data_skew_table_partitioned, org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, [id#6L, task_pod_name#7, project_id#8L, create_time#9, plan_type#10, deleted#11], [ds#12]" }],

可以提供完整的一点eventlog?或者dag graph截图?

zebozhuang avatar Sep 19 '23 08:09 zebozhuang

spark event log日志文件 application_1693822918548_1324.log

ketingli1 avatar Sep 19 '23 08:09 ketingli1

application_1693822918548_1324.log

我们看下

zebozhuang avatar Sep 19 '23 08:09 zebozhuang

您好,有结论吗

ketingli1 avatar Sep 20 '23 01:09 ketingli1

您好,有结论吗

这个应该是另个扫描表节点,没有包括在内的, 应该是要支持起来的,你们可以提个pull request补充这个逻辑吗?

zebozhuang avatar Sep 20 '23 01:09 zebozhuang

您好,有结论吗

另外,抱歉,昨天加错解析json包,比较晚看到结果。这个是通过个spark submit提交的任务吗?看解析plan里面的table可能需要小调整

zebozhuang avatar Sep 20 '23 02:09 zebozhuang

对,是spark submit提交的,还需要我提个pull request吗?

ketingli1 avatar Sep 21 '23 06:09 ketingli1

对,是spark submit提交的,还需要我提个pull request吗?

可以的,非常欢迎

zebozhuang avatar Sep 21 '23 08:09 zebozhuang