Does the new spark job type work with DSE?

[vikramkone]

I see that the spark job type is now available in the plugins. Kudos! Does this work with datastax version of cassandra where we submit spark jobs using DSE tools as follows ?

dse -u name -p * spark-submit --class org.ofe.App ./table_mon-1.0.jar

[johnyu0520]

Hey Vikram, What the code does is to feed the parameters into SparkSubmit class's main method. From your syntax, it seems like dse is another wrapper on top of spark-submit? I don't think it will work with the current implementation :(

Thanks, John

[vikramkone]

That's what I thought. Thanks for confirming On Jan 20, 2016 10:15 PM, "johnyu0520" [email protected] wrote:

Hey Vikram, What the code does is to feed the parameters into SparkSubmit class's main method. From your syntax, it seems like dse is another wrapper on top of spark-submit? I don't think it will work with the current implementation :(

Thanks, John

— Reply to this email directly or view it on GitHub https://github.com/azkaban/azkaban-plugins/issues/197#issuecomment-173469566 .