lifulong

Results 8 comments of lifulong

Organization: Zhihu Inc Region: Beijing, China Contact: [[email protected]](mailto:[email protected]) Usage Scenario: We use Apache Celeborn to improve the stability and efficiency of Spark engine shuffle. ![image](https://github.com/apache/incubator-celeborn/assets/450539/e25f150d-9895-481e-89a2-38151c0707ec)

c++ standerd 17 not support register key word, this key word has no effect in new c++ standerd temp solution we use is edit ep/build-velox/src/build_velox.sh and add -Wno-error=register compile option,...

i hava fix this issue at https://github.com/facebookincubator/velox/pull/12176, can you help review the fix? @Yuhta @FelixYBW @zml1206

> DId you try `spark.gluten.sql.columnar.shuffle.celeborn.useRssSort=false` ? our gluten version is 1.4, has not this conf yet

> Could we try this case with version 1.5.0? It looks like there’s a fix for issue [#9163](https://github.com/apache/incubator-gluten/issues/9163) — could you check if it works for you? our shuffle partitions...

i will test with set spark.celeborn.client.spark.shuffle.writer to rss_sort first

> i will test with set spark.celeborn.client.spark.shuffle.writer to rss_sort first set spark.celeborn.client.spark.shuffle.writer to sort, shuffle data is small a lot, performance optimize 10%, but still slow than vanilla spark