Do all the nodes of hadoop yarn cluster need to install CaffeOnSpark ?

[guyang88]

@mriduljain @davglass @gyehuda @javadba @pcnudde I want to run CaffeOnSpark on Spark-on-Yarn Cluster, do i need to install CaffeOnSpark on every node, or just install on one of them?

[junshi15]

You can either install it on each node, or install it on the node where you launch your job and use spark-submit to ship the whole package to executors. In either case, path needs to be set up properly.

[guyang88]

@junshi15 I installed CaffeOnSpark on one node and set the path per GetStarted_yarn , when I used spark-sbumit to launch caffeonspark ,I met mistake:"no lmdbjni in java path". But I installed caffeonspark on all node , I succeed. so what is the problem？(I have set LD_LIBRARY_PATH before spark-submit)

[anfeng]

You should create a tgz file, say cos.tgz, with lib64/liblmdbjni.so etc, and specify that tgz file as --archive and extend executor's LD_LIBRARY_PATH to include ":cos.tgz/lib64".

tar -cpzf ${HOME}/tmp/cos.tgz lib64

spark-submit .... --archives ${HOME}/tmp/caffe_on_grid_archive.tgz
--conf spark.driver.extraLibraryPath="WHATEVRYOUHAVE:./cos.tgz/lib64"
--conf spark.executorEnv.LD_LIBRARY_PATH="WHATEVRYOUHAVE:./cos.tgz/lib64" \

Andy

On Tue, Apr 4, 2017 at 7:18 PM, guyang88 [email protected] wrote:

I installed CaffeOnSpark on one node and set the path per GetStarted_yarn , when I used spark-sbumit to launch caffeonspark ,I met mistake:"no lmdbjni in java path". But I installed caffeonspark on all node , I succeed. so what is the problem？(I have set LD_LIBRARY_PATH before spark-submit)

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/yahoo/CaffeOnSpark/issues/241#issuecomment-291718908, or mute the thread https://github.com/notifications/unsubscribe-auth/AClTeEKysQCqara09xgAXZKt1JfvS8Qmks5rsvnogaJpZM4MvLza .

[guyang88]

@anfeng Thangks for your answer . I created a tgz file including CaffeOnSpark and all the lib*.so which are requierd, then I used spark-sbumit --archives / path to my tgz file . But I met the same error :"no lmdbjni in java path" .How can I make the tgz file shared by other nodes ? Do I create a fake tgz file?

[junshi15]

The .tgz file will be shipped to all executors. Make sure

1. your .tgz file contains lmdbjni.so
2. set the library path as shown by @anfeng