[GLUTEN-7685][VL][RAS] Optimize rough model
What changes were proposed in this pull request?
(Fixes: #7685) In production, we encountered a situation where the consumption of r2c large table was far greater than the performance improvement brought by native. for example:
sql("set spark.gluten.sql.columnar.filescan=false")
spark.range(100000000).toDF("id").selectExpr("concat('id_', round(id/1000000)) as k", "id % 10 as v")
.write.mode("overwrite").parquet("tmp/t1")
spark.read.parquet("tmp/t1").createOrReplaceTempView("t1")
sql("select k,sum(v) as v from t1 group by k").collect()
The local test takes 18 seconds to enble gluten, and only 6 seconds to disablegluten.Therefore, I hope to fallback this through RAS.
The optimization points are as follows:
- Increase bytesSize factor, cost = bytesSizeFactor * opCost
- r2c cost can be configured separately, and the default is 100. If sizeBytes is less than the threshold, the cost of RowToColumnarLike is ignored.
- Remove r2c fallback containing complex types
- Vanilla op cost is configurable, the default is 20, gluten op cost is 1
How was this patch tested?
https://github.com/apache/incubator-gluten/issues/7685
Run Gluten Clickhouse CI
Run Gluten Clickhouse CI
Run Gluten Clickhouse CI
Run Gluten Clickhouse CI
CI failed unrelated.
Run Gluten Clickhouse CI
Run Gluten Clickhouse CI
Run Gluten Clickhouse CI
Is it possible for you to share some test results about this change? Thanks!
I ran it in our grayscale adhoc query environment yesterday, and executorCpuTime overall reduced by 10%. @zhztheplayer
Run Gluten Clickhouse CI
Is it possible for you to share some test results about this change? Thanks!
I ran it in our grayscale adhoc query environment yesterday, and executorCpuTime overall reduced by 10%. @zhztheplayer
Sounds great.
BTW, based on my impression RoughCostModel has been used by a few of users. If you want to continue working on the new costers, would you consider creating a new one like RoughCostModel2, with alias rough2, probably? So we don't break the current usages of rough model by accident. Once it gets mature enough, we can move the logics to a more standardized cost model then.
Run Gluten Clickhouse CI
@zml1206 Can you add one line in PR description to demonstrate the way to enable this cost model? Thanks.
@zml1206 Can you add one line in PR description to demonstrate the way to enable this cost model? Thanks.
Added.