advancedxy

@szehon-ho Thanks for taking #9661 over and sorry for the late response. I was busy finishing a big internal feature last week. I extracted the API/Core part from previous POC...

Is there a design doc for this one? I can help review this feature.

You mean map side combine? Well, last time I checked, uniffle don't do any combine, just create a combined value for each record. If we are going to support this...

> I found shuffleMetaMap in LocalStorageMeta is useless, maybe we should remove it. And then we can remove LocalStorageMeta at the same time. If LocalStageMeta is removed, how to support...

> shuffleMetaMap is useless, right? I don't think ShuffleMetaMap is useless. I believe its intention is to track every shuffleId's shuffle size and last read time. It should be updated...

I wrote up a design doc for this issue: https://docs.google.com/document/d/1OGswqDDQ52rpw5Lat-FpEGDfX1T6EFXYLC7L0G5O4WE/edit?usp=sharing @zuston @xianjingfeng @jerqi could you mind do some design review?

> @advancedxy I have some comments in the design doc. Please grant me 'comment' permission. Changed the default permission from viewer to commenter. Please try to refresh it and check...

>If it's ok, I will go ahead. Gently ping. @jerqi @zuston @kaijchen @advancedxy I think cli is a good way to manage shuffle clusters. I create a cli support ticket...

> > > If it's ok, I will go ahead. Gently ping. @jerqi @zuston @kaijchen @advancedxy > > > > > > I think cli is a good way to...

> If the range of [low-watermark, high-watermark] is large, this problem with this patch should still exist, right? +1. I'm not sure how your strategy to choose partitions over shuffles...