zipkin-sparkstreaming icon indicating copy to clipboard operation
zipkin-sparkstreaming copied to clipboard

Published 20 hours ago • verified publisher icon openzipkin-attic

Document using with YARN

Open devinsba opened this issue 8 years ago • 2 comments

I'll preface this by saying I have zero operational knowledge about spark so...

In order to use this in AWS with EMR we will need to be able to connect to a YARN controlled Spark cluster. I'm not sure if currently the project even supports YARN but it would be good to know and have some documentation around if/when it does.

devinsba avatar Mar 08 '17 19:03 devinsba

looks like normally yarn is a different distribution, which special cases the magic value "yarn" as the cluster name. It might be possible to hack our main class to do the same, or we document using spark-submit, treating this app as a normal jar.

here are some related links

https://spark.apache.org/docs/1.6.3/running-on-yarn.html http://blog.sequenceiq.com/blog/2014/08/22/spark-submit-in-java/

codefromthecrypt avatar Mar 09 '17 13:03 codefromthecrypt

EDIT: jinx again.. 😄

devinsba avatar Mar 09 '17 13:03 devinsba