risingwave icon indicating copy to clipboard operation
risingwave copied to clipboard

Published 20 hours ago • verified publisher icon risingwavelabs

opt(source): Source performance does not seem to scale linearly with cores

Open jon-chuang opened this issue 3 years ago • 2 comments
trafficstars

Source performance seems not to scale linearly with 6 threads v.s. 12 threads with materialized view turned off. It could be due to SMT as the bench was run on 16vCPU x86 machine. We should test source throughput of 4T, 6T, 8T, 12T and 16T, 24T on a 32vCPU machine to see if there may be some non-linearity if SMT is not interfering.

CC @lmatz

jon-chuang avatar Sep 20 '22 10:09 jon-chuang

Which kind of source did you test? 🤔

BugenZhao avatar Sep 20 '22 10:09 BugenZhao

not Kafka source? It's the computation part that does not scale.

lmatz avatar Sep 20 '22 11:09 lmatz

This issue has been open for 60 days with no activity. Could you please update the status? Feel free to continue discussion or close as not planned.

github-actions[bot] avatar Nov 20 '22 02:11 github-actions[bot]

any update?

tabVersion avatar Nov 20 '22 05:11 tabVersion

No updates for now, we will continue to investigate alongside performance of Nexmark queries. Maybe this is better tracked in the performance-related docs. So I will close it for now.

jon-chuang avatar Nov 21 '22 08:11 jon-chuang

If you mean the nexmark datagen source, I think @wangrunji0408 is working on the optimization now. 🚀 https://github.com/risingwavelabs/risingwave/issues/4555

BugenZhao avatar Nov 21 '22 08:11 BugenZhao