rubix icon indicating copy to clipboard operation
rubix copied to clipboard

Published 20 hours ago • verified publisher icon qubole

Rubix + Spark Usage instructions

Open vrajat opened this issue 6 years ago • 1 comments

Current README has the following instructions

Add the Rubix jars to Driver and Executor classpaths and use the rubix implementation for hadoop.fs.s3 and hadoop.fs.s3n. --conf spark.hadoop.fs.s3.impl=com.qubole.rubix.hadoop2.CachingNativeS3FileSystem --conf spark.hadoop.fs.s3n.impl=com.qubole.rubix.hadoop2.CachingNativeS3FileSystem --conf spark.driver.extraClassPath=/usr/lib/hadoop2/share/hadoop/common/lib/rubix-core-${rubix-version}.jar,/usr/lib/hadoop2/share/hadoop/common/lib/rubix-hadoop2-${rubix-version}.jar,/usr/lib/hadoop2/share/hadoop/common/lib/rubix-bookkeeper-${rubix-version}.jar --conf spark.executor.extraClassPath=/usr/lib/hadoop2/share/hadoop/common/lib/rubix-core-${rubix-version}.jar,/usr/lib/hadoop2/share/hadoop/common/lib/rubix-hadoop2-${rubix-version}.jar,/usr/lib/hadoop2/share/hadoop/common/lib/rubix-bookkeeper-${rubix-version}.jar Start/Re-start the Spark Application.

We should consider coding this up in rubix-admin if possible

vrajat avatar May 03 '18 03:05 vrajat

Hello. I don't see anything related to Spark now in Readme. Where I can find these instructions?

deema0 avatar Dec 12 '18 18:12 deema0