CloudShuffleService
CloudShuffleService copied to clipboard
bytedance
Cloud Shuffle Service(CSS) is a general purpose remote shuffle solution for compute engines, including Spark/Flink/MapReduce.
### What would you like to be added? I am trying to build a CSS docker image to support Spark3.2 based on java11 (FROM amazoncorretto:11 ), not able to compile...
1. if I use all default settings, and start as `sbin/start-all.sh`, a Worker and a Master are running but when I submit a Spark app, it throw `Caused by: java.lang.RuntimeException:...
1. Does it support both remote shuffle service and dynamic resource allocation? 2. Does the enabling of `spark.shuffle.service.enabled` have an impact on the shuffle service?
#### What type of PR is this? enhancements：reduce driver memory usage #### What this PR does / why we need it: When running large data task，spark driver offen encounters oom...
spark 分区数据量特别大20多万的分区，driver内存为15G，由于master是通过driver启动的，作业运行1/3之后，shuffleEpochSetMap 占用内存将近10G左右，导致driver内存溢出，shuffleEpochSetMap 的大小跟什么有关系，为什么会占用这么多的内存，有什么解决办法吗
It is introduced that CSS supports Flink Shuffle, but this part of the feature is not in the code. Is there any plan to support Flink Shuffle in the future?
### What happened? After I configure it in accordance with the readme document, the error is reported through the instruction**. / sbin/start-worker.sh** startup is as follows **. / sbin/start-workers.sh or...
#### What type of PR is this? enhancements #### Special notes for your reviewer: Dear repository maintainers, I am developing a model designed to automatically generate concise and accurate summaries...
