RemoteShuffleService icon indicating copy to clipboard operation
RemoteShuffleService copied to clipboard

Published 20 hours ago • verified publisher icon uber

Shuffle Files Storage Is stored by default.Whether alluxio storage is supported and how to implement it.

Open liangrui1988 opened this issue 2 years ago • 5 comments

Shuffle Files Storage Is stored by default.Whether alluxio storage is supported and how to implement it.

liangrui1988 avatar May 12 '22 06:05 liangrui1988

RSS currently only supports local storage. Have a look at com.uber.rss.storage.ShuffleStorage for plugging in other storage

mayurdb avatar May 12 '22 07:05 mayurdb

Thank you. One more question.Spark. Shuffle. RSS. DataCenter = dc1 the dc1 is to point to?Does local storage mean disk storage?How do you specify multiple disk directories?

liangrui1988 avatar May 12 '22 11:05 liangrui1988

@liangrui1988 i'm considering trying to add this support to RemoteShuffleService, would you be interested for me to try and contribute it to this repo?

cpd85 avatar May 23 '22 20:05 cpd85

@liangrui1988 i'm considering trying to add this support to RemoteShuffleService, would you be interested for me to try and contribute it to this repo?

Yeah, you are welcome to contribute!

hiboyang avatar May 26 '22 05:05 hiboyang

Thank you. One more question.Spark. Shuffle. RSS. DataCenter = dc1 the dc1 is to point to?Does local storage mean disk storage?How do you specify multiple disk directories?

"DataCenter = dc1" is just a tag to distinguish different remote shuffle service instance, you could just leave it with default value dc1 or any string value.

Yes, local storage means local disk storage.

It does not support multiple disk directories right now. But feel free to add a PR for that.

hiboyang avatar May 26 '22 05:05 hiboyang