Jun Shi comments

Results 115 comments of


                                            Jun Shi

image dataset on HDFS

CaffeOnSpark does not read individual image files (although caffe does). The images need to be saved in some spark-friendly formats. Data-frame is one of them. You can think of data-frame...

Support for Protocol buffers > 2.5

There is no plan to upgrade protobuf at this moment. Is there any reason you prefer 2.6?

Make build error:Failed to execute goal org.scalatest:scalatest-maven-plugin:1.0

Some unit tests failed? We will know more if you can show more log.

Make build error:Failed to execute goal org.scalatest:scalatest-maven-plugin:1.0

InterleaveTest failed. I am not sure why, though.

Error when building [exec] src/main/cpp/jni/JniMat.cpp: In function ‘void Java_com_yahoo_ml_jcaffe_Mat_decode(JNIEnv*, jobject, jint, jlong)’: [exec] src/main/cpp/jni/JniMat.cpp:82:65: error: too many arguments to function ‘cv::Mat cv::imdecode(cv::InputArray, int)’

Your openCV version may not be compatible. We use openCV2.4, which is not compatible with 3.x.

What's the difference and relation between --executor-cores and -devices in CPU mode?

-devices is number of GPU devices per executor. it has no effects in the CPU mode.

CaffeOnSpark CPU model failed but GPU model success(same scripts and same data)

Did you turn on the test phase? if so, change the solver.prototxt to turn it off. test_iter: 0 test_interval: 0

Use infiniband

from your ibstat log: Port 1: State: Down Your port is down. Please get a local expert to help you with Infiniband adapters, verify your connection is correct, before you...

error: ‘bp’ does not name a type

remove caffe_mini_cluster.cpp. That file is broken. You don't need it to run CaffeOnSpark.

There are some problems when training the example

Please use "yarn logs -applicationId" to get more detailed executor log.