blaze icon indicating copy to clipboard operation
blaze copied to clipboard

Supports apache-celeborn

Open richox opened this issue 2 years ago • 2 comments

Is your feature request related to a problem? Please describe. apache-celeborn is widely used as an RSS (remote shuffle service) for spark. currently blaze has supported an internal RSS implementation from Kuaishou.Inc, however celeborn is not yet supported.

Describe the solution you'd like integrate blaze with celeborn, this should be done by extending rss-xxx-repartitioner.rs. furthermore, we can support an individual rss shim module for supporting different RSS implementations.

Describe alternatives you've considered

Additional context

richox avatar Nov 03 '23 07:11 richox

Uniffle is another candidate. Do you mind integrating uniffle with blaze?

zuston avatar Nov 03 '23 07:11 zuston

Uniffle is another candidate. Do you mind integrating uniffle with blaze?

i suggest we create a abstract layer for integrating different rss implementations. of course we can have one working celeborn/uniffle module, then we can have more idea on designing the abstract layer.

richox avatar Nov 06 '23 17:11 richox