Getting jar from CAPS project in intellije and Running it in cluster

[MohamedRagabAnas]

Hi, I want to run specific example from CAPS project on another machine how to extract the jar file in the CAPS project (should it be a fat jar).

then how to run it that machine will it be submitted as a spark job using 'spark-submit' or how to run it in the right way

Thanks in advance

[Mats-SX]

Hi @MohamedRagabAnas

Just run ./gradlew allJar -x test from the root directory, which should give you a morpheus-spark-cypher-all.jar file in morpheus-spark-cypher/build/libs.

[MohamedRagabAnas]

Thanks @Mats-SX, How to specify a specific scala to be run. and How to run the output jar is it using spark-submit ? or java -jar ?

[Mats-SX]

This depends on how you've set up your Spark cluster. Generally, using spark-submit is recommended.

https://spark.apache.org/docs/latest/submitting-applications.html

[MohamedRagabAnas]

Thank You and How building the CAPS with multiple main classes. how to specify main class to be run.

[MohamedRagabAnas]

Task 'allJar' not found in root project 'okapi'. Some candidates are: 'docJar', 'jar', 'jmhJar'.

[Mats-SX]

That's odd. When I run ./gradlew allJar -x test from the root directory I get the expected result. Are you sure you're running exactly that command?

In order to specify main class, please refer to the standard Spark documentation. We don't do anything in particular but the standard approaches should be enough.

[MohamedRagabAnas]

Thanks @Mats-SX I figured out the problem was with running the command and older version of Morpheus. but according to the main class I don't mean how to specify the main class in the spark-submit in spark but how to specify it in the Morpheus build.gradle using the jar{manifest: mainclass:...}.

I'm just confused about this point if you just have the morpheus-spark-cypher-all.jar with multiple classses inside how to specify which one will be submitted or run as a job?

[MohamedRagabAnas]

Ok Mats I have figured this out by putting the .all Jar file into spark and running examples jar by just giving the name of the class I want to run.

[zhengminlai]

Ok Mats I have figured this out by putting the .all Jar file into spark and running examples jar by just giving the name of the class I want to run.

Can you tell me what's the command for running examples using .all jar file? I use spark-submit --class org.opencypher.morpheus.examples.CaseClassExample --master local morpheus-spark-cypher-0.3.3-SNAPSHOT-all.jar but it turns out ClassNotFound Error. Thanks.

[zhengminlai]

Ok Mats I have figured this out by putting the .all Jar file into spark and running examples jar by just giving the name of the class I want to run.

Can you tell me what's the command for running examples using .all jar file? I use spark-submit --class org.opencypher.morpheus.examples.CaseClassExample --master local morpheus-spark-cypher-0.3.3-SNAPSHOT-all.jar but it turns out ClassNotFound Error. Thanks.

@Mats-SX @MohamedRagabAnas Can u please help me with this? It's very urgent:(

[Mats-SX]

Hello @zhengminlai

The examples are not included in the morpheus-spark-cypher-*-all jar. This jar contains only the necessary code to develop Morpheus applications (such as the examples). In order to execute the examples, you will need to put both the all jar as well as the morpheus-examples jar on the classpath:

bin/spark-submit --master local --class org.opencypher.morpheus.examples.CaseClassExample --jars morpheus-spark-cypher-0.4.1-SNAPSHOT-all.jar morpheus-examples-0.4.1-SNAPSHOT.jar

[MohamedRagabAnas]

@Mats-SX the new version of Morpheus doesn't support ./gradlew allJar -x test command to get the *all.jar file and turns up this error: Task 'alljar' not found in root project 'okapi'. Some candidates are: 'docJar', 'jar', 'jmhJar'.