incubator-uniffle icon indicating copy to clipboard operation
incubator-uniffle copied to clipboard
verified publisher icon apache

Benchmark: ESS and Uniffle

Open zuston opened this issue 3 years ago • 4 comments

Could the community provide the official benchmark with ESS and uniffle?

zuston avatar Jul 26 '22 11:07 zuston

@zuston You can refer this blog for the benchmark related.

colinmjj avatar Jul 28 '22 03:07 colinmjj

Attach the google doc about test result in our internal cluster： https://docs.google.com/document/d/1nmHMBEaa4lHfgQkdlYokTtXt12F5vRJiQ2MCmTQbW6k/edit#heading=h.b1udpb9l28w7

I can't achieve the same result from the blog. Could u help check this? @colinmjj @jerqi

zuston avatar Jul 28 '22 03:07 zuston

@zuston The benchmark of blog is based on Spark 2.4.6. If there has no random disk IO problem with ESS, Uniffle is expected has poor performance than ESS

colinmjj avatar Jul 28 '22 03:07 colinmjj

Yes, you are right. @colinmjj . I tested in 5TB terasort, ESS directly failed when shuffle read due to 100% diskio. But uniffle looks good.

I have updated latest result in google doc. @colinmjj @jerqi . I think maybe it would be better to attach the result to github issue.

zuston avatar Jul 28 '22 11:07 zuston