splash
splash copied to clipboard
MemVerge
Splash, a flexible Spark shuffle manager that supports user-defined storage backends for shuffle data storage and exchange
Bumps hadoop-common from 2.7.4 to 3.2.3. [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a...
![dependabot[bot] avatar](https://avatars.githubusercontent.com/in/29110?v=4 "dependabot[bot] avatar"){kind=avatar}
Hello, I have built and set up splash and spark according to the documentation. But when I try to run any spark application I am getting the following error message:...
Bumps commons-io from 2.4 to 2.7. [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a...
In cloud environments, it is a common requirement to be able to persist shuffle data outside of the node on which a Spark task is running. Since many workloads run...
Allow the plugin developer to implement a fast merge which will be invoked when: * Encryption is disabled. * Compression is disabled or the compression codec supports concatenation of serialized...
Before build I've changed spark version in pom.xml: Spark-submit done in local[*] mode 2.4.0 java.lang.IllegalAccessError: tried to access class org.apache.spark.shuffle.sort.ShuffleInMemorySorter from class org.apache.spark.shuffle.sort.SplashUnsafeSorter at org.apache.spark.shuffle.sort.SplashUnsafeSorter.(SplashUnsafeSorter.scala:77) at org.apache.spark.shuffle.sort.SplashUnsafeShuffleWriter.(SplashUnsafeShuffleWriter.scala:56) at org.apache.spark.shuffle.SplashShuffleManager.getWriter(SplashShuffleManager.scala:84) at...
The output of the shuffle performance tool is not precise enough. For example, 1.23GB would only be displayed as 1GB. The output should keep at least two digits for the...