John Cherian

icon company Amazon Web Services icon location DC-Area icon location AWS and Spark Architect

Results 13 comments of John Cherian

Include S3 based Shuffle storage plugin

@gauravtanwar03 Good thinking. Do you have a prototype ready?

Include S3 based Shuffle storage plugin

@gauravtanwar03 do you have a working code ?

Include S3 based Shuffle storage plugin

We can include .config("spark.shuffle.spill", True) \ .config("spark.shuffle.spill.compress", True) \ .config("spark.shuffle.compress", True) \ .config("spark.shuffle.file.buffer", "64k") \ .config("spark.local.dir", "s3a://your-bucket-name/spark/shuffle/") \