sparkmagic icon indicating copy to clipboard operation
sparkmagic copied to clipboard

Published 20 hours ago • verified publisher icon jupyter-incubator

Create official Docker images (was: docker-compose build failure)

Open devender-yadav opened this issue 5 years ago • 17 comments 

docker-compose build

Not able to build spark core.

Error logs:

[error] /apps/build/spark/core/src/test/java/test/org/apache/spark/JavaAPISuite.java:36: warning: [deprecation] Accumulator in org.apache.spark has been deprecated [error] import org.apache.spark.Accumulator; [error] ^ [error] /apps/build/spark/core/src/test/java/test/org/apache/spark/JavaAPISuite.java:37: warning: [deprecation] AccumulatorParam in org.apache.spark has been deprecated [error] import org.apache.spark.AccumulatorParam; [error] ^ [error] Compile failed at Jul 4, 2019 8:47:28 AM [1:25.010s] [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary: [INFO] [INFO] Spark Project Parent POM ........................... SUCCESS [05:03 min] [INFO] Spark Project Tags ................................. SUCCESS [ 35.560 s] [INFO] Spark Project Sketch ............................... SUCCESS [ 4.462 s] [INFO] Spark Project Local DB ............................. SUCCESS [ 20.445 s] [INFO] Spark Project Networking ........................... SUCCESS [01:03 min] [INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [ 5.563 s] [INFO] Spark Project Unsafe ............................... SUCCESS [ 24.045 s] [INFO] Spark Project Launcher ............................. SUCCESS [01:39 min] [INFO] Spark Project Core ................................. FAILURE [05:06 min] [INFO] Spark Project ML Local Library ..................... SUCCESS [ 38.587 s] [INFO] Spark Project GraphX ............................... SKIPPED [INFO] Spark Project Streaming ............................ SKIPPED [INFO] Spark Project Catalyst ............................. SKIPPED [INFO] Spark Project SQL .................................. SKIPPED [INFO] Spark Project ML Library ........................... SKIPPED [INFO] Spark Project Tools ................................ SUCCESS [ 34.015 s] [INFO] Spark Project Hive ................................. SKIPPED [INFO] Spark Project REPL ................................. SKIPPED [INFO] Spark Project YARN Shuffle Service ................. SUCCESS [ 31.793 s] [INFO] Spark Project YARN ................................. SKIPPED [INFO] Spark Project Hive Thrift Server ................... SKIPPED [INFO] Spark Project Assembly ............................. SKIPPED [INFO] Spark Integration for Kafka 0.10 ................... SKIPPED [INFO] Kafka 0.10 Source for Structured Streaming ......... SKIPPED [INFO] Spark Project Examples ............................. SKIPPED [INFO] Spark Integration for Kafka 0.10 Assembly .......... SKIPPED [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 12:26 min (Wall Clock) [INFO] Finished at: 2019-07-04T08:47:28+00:00 [INFO] Final Memory: 57M/764M [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal net.alchim31.maven:scala-maven-plugin:3.2.2:testCompile (scala-test-compile-first) on project spark-core_2.11: Execution scala-test-compile-first of goal net.alchim31.maven:scala-maven-plugin:3.2.2:testCompile failed. CompileFailed -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginExecutionException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn -rf :spark-core_2.11 ERROR: Service 'spark' failed to build: The command '/bin/sh -c mkdir -p /apps/build && cd /apps/build && git clone https://github.com/apache/spark.git spark && cd $SPARK_BUILD_PATH && git checkout v$SPARK_BUILD_VERSION && dev/make-distribution.sh --name spark-$SPARK_BUILD_VERSION -Phive -Phive-thriftserver -Pyarn && cp -r /apps/build/spark/dist $SPARK_HOME && rm -rf $SPARK_BUILD_PATH' returned a non-zero code: 1

devender-yadav avatar Jul 04 '19 09:07 devender-yadav

I cannot seem to reproduce this. Are you building from the latest master?

apetresc avatar Jul 05 '19 18:07 apetresc

@apetresc yes from the master? Did you try building the image again?

devender-yadav avatar Jul 05 '19 18:07 devender-yadav

Yes, I just tried again with docker-compose build --no-cache spark to make sure I didn't have some cached layer containing a different Java version, and it still worked fine:

Successfully built 9a115fae44fc 
Successfully tagged jupyter/sparkmagic-livy:latest

Can you try with the cache disabled, just to try and rule that out?

apetresc avatar Jul 05 '19 18:07 apetresc

@apetresc I'll try tomorrow. I am wondering if I can pull it from dockerhub directly. I am suggesting AWS to update sparkmagic version for EMR notebook. I just wanna try the latest version once at my side first.

devender-yadav avatar Jul 05 '19 18:07 devender-yadav

@itamarst can we update sparkmagic image to dockerhub. I don't know why but I am still getting errors in spark code compilation in docker compose build step.

devender-yadav avatar Jul 09 '19 01:07 devender-yadav

@devender-yadav which image are you referring to exactly? What name? (I'm new to the project, so don't know what images there are or how they're built).

itamarst avatar Jul 12 '19 17:07 itamarst

This builds fine for me too when I do --no-cache on master.

itamarst avatar Jul 12 '19 18:07 itamarst

@itamarst It's failing while building Dockerfile.spark in the below-mentioned step:

RUN mkdir -p /apps/build && \
  cd /apps/build && \
  git clone https://github.com/apache/spark.git spark && \
  cd $SPARK_BUILD_PATH && \
  git checkout v$SPARK_BUILD_VERSION && \
  dev/make-distribution.sh --name spark-$SPARK_BUILD_VERSION -Phive -Phive-thriftserver -Pyarn && \
  cp -r /apps/build/spark/dist $SPARK_HOME && \
  rm -rf $SPARK_BUILD_PATH

devender-yadav avatar Jul 14 '19 07:07 devender-yadav

Same issue: [ERROR] Failed to execute goal net.alchim31.maven:scala-maven-plugin:3.2.2:compile (scala-compile-first) on project spark-graphx_2.11: Execution scala-compile-first of goal net.alchim31.maven:scala-maven-plugin:3.2.2:compile failed. CompileFailed when run docker build -t sparkmagic-livy -f ./Dockerfile.spark .

ragelo avatar Aug 05 '19 12:08 ragelo

Fascinating.

@ragelo

  1. Is this first time you've ever built this image, or have you done so before on this machine?
  2. Can you include output of git log -1.

itamarst avatar Aug 05 '19 12:08 itamarst

@itamarst

  1. Yes, it's the first time for this image (I also did 3-4 retries after).
commit ee092134588287436607d6c7197ba61b4e3856f3 (HEAD -> master, origin/master)
Merge: b771129 767c581
Author: Itamar Turner-Trauring <[email protected]>
Date:   Fri Jul 19 14:33:14 2019 -0400

    Merge pull request #555 from jupyter-incubator/0.12.9
    
    Prepare for 0.12.9.

ragelo avatar Aug 05 '19 13:08 ragelo

Oh, I wonder if I couldn't reproduce it because I did --no-cache but I didn't also do --pull. Will investigate later this week.

itamarst avatar Aug 05 '19 15:08 itamarst

So I just did:

$ sudo docker build --no-cache --pull -t jupyter/sparkmagic-livy -f Dockerfile.spark .

And it worked fine. So I have no idea what's up.

@devender-yadav you mentioned dockerhub, do you know of an official docker image? If not I can upload one.

itamarst avatar Sep 05 '19 20:09 itamarst

@itamarst I don't see any official image at dockerhub. On searching sparkmagic, I found https://hub.docker.com/r/heliumdatacommons/sparkmagic but this is updated year back and not sure if it is reliable.

It's good to keep the latest image at dockerhub so that we can directly pull from there.

devender-yadav avatar Sep 06 '19 05:09 devender-yadav

I am having this issue - but it seems to fail at a different step. This is the first time I've ever built this image on this machine.

  • git log -1 says
commit 12fc2a6dc49f21a93884ce3d0bd0f842118ad125 (HEAD -> master, origin/master, origin/HEAD)
Merge: 47aeb09 83dd808
Author: Itamar Turner-Trauring <[email protected]>
Date:   Thu Oct 3 20:03:32 2019 -0400

    Merge pull request #577 from devstein/patch-1
    
    Use JSON true instead of Python True in README Papermill Section
  • The error happens when executing $ sudo docker build --no-cache --pull -t jupyter/sparkmagic-livy -f Dockerfile.spark . with the message
Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.2.1:exec (python-api install) on project livy-python-api

I am attaching the log from that build command.

dockerbuild-spark.log

ibiris avatar Nov 06 '19 11:11 ibiris

Relates to #756

devstein avatar Apr 27 '22 19:04 devstein

Is there a plan to push sparkmagic image to some Docker registry?

insider89 avatar Apr 11 '23 08:04 insider89