RemoteShuffleService icon indicating copy to clipboard operation
RemoteShuffleService copied to clipboard

close writers during end of stage using onStageCompleted

Open tuckerh99 opened this issue 2 years ago • 4 comments

Improvement on when to close writers. While implementing an s3 storage engine, noticed that we want to close the writers sooner than later to ensure shuffle files are stored on s3 by the end of a stage.

tuckerh99 avatar Aug 04 '22 18:08 tuckerh99

CLA assistant check
All committers have signed the CLA.

CLAassistant avatar Aug 04 '22 18:08 CLAassistant

@tuckerh99, thanks for the PR! Curious whether you run your Spark production workload with shuffle files storing on S3? If so, how is the performance?

hiboyang avatar Aug 25 '22 16:08 hiboyang

@hiboyang hi we are testing this out actively right now. in the first iteration it seems the performance impact is very high due to the nature of shuffle data. Basically if we wait to merge the shuffle files until the stage is completed, then we need to add in some delay before starting the next stage while the shuffle data is uploaded to S3.

cpd85 avatar Aug 29 '22 19:08 cpd85

Hi @tuckerh99 how are things going on the later iterations? Any luck?

leletan avatar Jan 22 '23 08:01 leletan