More of a question than an issue.. what IDE/env is used by committers to do SparkNet development?

[javadba]

(note: Please move this "issue" to an appropriate thread/location/etc.. I can't find any gitter or mailing list then email me info javadba at gmail.com)

I am joining a project performing distributed GPU based work on SparkNet. I am a former contributor to Spark (sql and mllib) so have strong underpinnings in the core spark stack.

It is crucial to productivity to have an IDE that at the least permits traversing the codebase. Respecting/ handling breakpoints would be a big plus.

Intellij is my preferred IDE but is not playing well with SparkNet. Here is one of the issues I am battling:

http://stackoverflow.com/questions/35385304/unable-to-import-sbt-project-into-intellij-using-custom-sbt-launch-jar

So then my question here is: what is the development environment and process for the core SparkNet committers? I want to be in sync with that in order to be able to be fully invested as a likely near-future contributor.

Thanks Stephen Boesch

[javadba]

FYI I have created a maven pom.xml that emulates the build.sbt and successfully builds the project. I am using this successfully within intellij - given the build.sbt is not accepted there. A PR for the pom.xml is forthcoming.

[robertnishihara]

Thanks Stephen, that's excellent! We've been using SBT so far because it's very simple, and we've been doing most of the development in Atom. We haven't defined a formal development environment/process yet, but we're open to suggestions about that.

[javadba]

https://github.com/amplab/SparkNet/pull/65

[javadba]

Atom is a good alternative: apropos I have been trying to get up to speed in it just this past two weeks. Are you using ensime? What other plugins? If you have link(s) (that can save you time from writing it yourself) to the stuff you use that's even better.

[javadba]

btw I am also going to be trying out scala 2.11 and Spark 1.6.0 / Spark 2.0 within the week and will send PR or create bug report if not possible.

[javadba]

It's cool to see fast turnaround on your comments. That's a nice warm fuzzy for starting on this project. Have you considered gitter?

[robertnishihara]

Thanks a lot! Definitely let us know if there are problems with the newer versions of Scala and Spark! We haven't quite figured out where the best place to have these discussions is, but Gitter is definitely an option.

We've been using Atom without any plugins, but Ensime looks like it'd be useful. Do you use it?

[javadba]

re:Ensime. I have not personally but my fanatical scala friend does and has found it useful. Will circle back in couple of weeks after leveraging (/stealing) his tricks of the trade there.

[robertnishihara]

Sounds good!

[javadba]

Unrelated: how to build the libccaffe? Here is what I did

sbt package assembly

Then

spark-submit --class apps.CifarApp target/scala-2.10/sparknet-assembly-0.1-SNAPSHOT.jar 1

However the libbcaffe.so is required. But it has not been built.

Caused by: java.lang.UnsatisfiedLinkError: Can't load library: /git/SparkNet/build/libccaffe.so

How to build that? I can not find any script or command to do so.

[robertnishihara]

Right now, we've been doing the following:

cd SparkNet
mkdir build
cd build
cmake ../libccaffe
make -j 30

Let me know if that works or doesn't work for you.

[javadba]

OK thanks Is there documentation (that I missed) that should be followed?

2016-02-14 11:17 GMT-08:00 Robert Nishihara [email protected]:

Right now, we've been doing the following:

cd SparkNet mkdir build cd build cmake ../libccaffe make -j 30

— Reply to this email directly or view it on GitHub https://github.com/amplab/SparkNet/issues/64#issuecomment-183957850.

[robertnishihara]

It's in the README under "setup". We probably should structure the README in a more intuitive way.

[javadba]

My bad. I did see it earlier. Got lost in the shuffle (setting up a bunch of stuff).

[javadba]

Unrelated question. What is the relationship between the caffe embedded in SparkNet at https://github.com/SparkNet/caffe and the standalone github repo @ https://github.com/BVLC/caffe ? Is the first one a snapshot of a specific version of the standalone? Are there any SparkNet specific changes?

[robertnishihara]

The caffe embedded in SparkNet is a snapshot of Caffe. We squashed it here https://github.com/amplab/SparkNet/commit/192ba1fa904bf1c956d74be992b655287ceebc7d.

We made some small changes, mostly related to multiGPUs, testing, and reading data from Java.

However, one of our big priorities right now is switching from our own Caffe wrapper to using javacpp-presets https://github.com/bytedeco/javacpp-presets (see #60). This will make things much easier to develop/deploy/maintain, and we've been working on this in the branch javacpp+dataframes. The main thing blocking us here is a bug with javacpp-presets (see bytedeco/javacpp-presets#147). We haven't had a chance to debug that yet. It's a key thing to do. One piece of low-hanging fruit is to figure out where in javacpp the bug was introduced.

To see the bug, you can look at the attached example. Calling sbt run from ExampleGPU/ should work, but if you uncomment the line Caffe.set_mode(Caffe.GPU), then the app will crash. Searching through the versions (and potentially commits) of javacpp-presets or maybe javacpp in build.sbt could give us an idea of where the bug was introduced. Any progress/ideas in that direction would be extremely helpful. If you're interested, we can also get in touch with the javacpp maintainers. ExampleGPU.zip

[javadba]

Thanks Robert I am in a bit over my head at this moment while wrestling with building on my Mac. I am getting partway through the caffe tests "make runtests" and likewise partway through the SparNet tests "sbt test". I have not reported anything about my challenges yet because they may be configuration errors on my part. I will circle back one way or other after getting more traction with the codebase and with the environment setup requirements.

[javadba]

Btw by using the maven / pom.xml with the PR for "specify master on commandline" I have been able to launch the CifarApp within Intellij including nice stuff like setting breakpoints. It is failing rather soon due to my inability (so far) to properly build libccaffe.

[javadba]

[robertnishihara]

Ah, some of the tests may not be working right now. We haven't been as diligent about that as we should have been. They mostly require small fixes, but I'd suggest not spending much time on that (you seem to be going through the apps, which is a better entry point).

What errors are you getting with building libccaffe? Caffe is a bit easier to build on Linux, but we've seen a number of errors, so maybe we can help. Fortunately, this kind of difficulty will go away once we switch to using JavaCPP.

[javadba]

Thanks - I am about to launch a full rebuild - this time with cudaDNN enabled - and go through the online docs once again to see if I missed some obvious stuff. Will get back tomorrow afternoon on that.

[javadba]

Here is the core of my challenges on mac: should we be using atlas or openblas? I have seen references to both of them. I am on El Capitan and currently using atlas - due to some comments about caffe from other projects.

The error I get on a bunch of files when building libccaffe is:

fatal error: 'cblas.h' file not found

Here is a full example

/git/sparknet/libccaffe/../caffe/include/caffe/util/mkl_alternate.hpp:11:10: fatal error: 'cblas.h' file not found

If there were a specific guide for building libccafe on mac that works best for this project would you please point me to it? Thanks!

[robertnishihara]

I think the real solution is to switch to JavaCPP, so people won't have to build libccaffe. Philipp and I will try to get this working tomorrow (at least without the GPU mode) so you don't have to worry about libccaffe.

Then we can remove the copy of Caffe from our repository and makes things much cleaner.

[javadba]

That's great! Thanks for the awesome responsiveness.

[javadba]

A bit different topic. I tried the javacpp+dataframes branch. I am running on mac. Do you have any suggestions how to approach the non-linux build?

ERROR] Failed to execute goal on project sparknet: Could not resolve dependencies for project org.amplab:sparknet:jar:1.0: The following artifacts could not be resolved: org.bytedeco.javacpp-presets:caffe:jar:macosx-x86_64:master-1.2-SNAPSHOT, org.bytedeco.javacpp-presets:opencv:jar:macosx-x86_64:3.1.0-1.2-SNAPSHOT, com.twelvemonkeys.imageio:imageio:jar:3.1.2, com.twelvemonkeys.imageio:common-lang:jar:3.1.2: Failure to find org.bytedeco.javacpp-presets:caffe:jar:macosx-x86_64:master-1.2-SNAPSHOT in https://repository.apache.org/content/repositories/snapshots was cached in the local repository, resolution will not be reattempted until the update interval of Maven snapshots repository has elapsed or updates are forced -> [Help 1]

[javadba]

Do you guys have a local (laptop/desktop) standardized environment? E.g. a docker image?

[robertnishihara]

These are both really good points.

1. We've been focusing on the Linux build. We haven't done much testing on other operating systems, but that's an important thing to do.

2. We also haven't made a docker image, but that would be useful. However, we can run things locally on top of a local Spark cluster. See some of the later discussion in #52 about this.

[robertnishihara]

Also, did you get the above error message when building with sbt or with maven? I actually put the files at http://www.eecs.berkeley.edu/~rkn/temp/org/bytedeco/ (this is not a permanent solution, but the current javacpp-presets release doesn't have some bug fixes that we need), and I added

resolvers += "javacpp" at "http://www.eecs.berkeley.edu/~rkn/temp/"

to the build.sbt.

[javadba]

re: .. ~rkn/temp/ .. I saw that - and had added to the within the maven build. I should have tried the sbt as well - will do that now.

[pcmoritz]

Another option (for now) is to use an older version of JavaCPP (version 1.1 should work), the downside is that the Solver won't work (it depends on a recent bugfix), but you could construct networks and run forward and backward methods on your Mac.