spark-deep-learning icon indicating copy to clipboard operation
spark-deep-learning copied to clipboard

sparkdl for using in Anaconda

Open fsahba opened this issue 7 years ago • 1 comments

Hello all I am wondering if there is any method to install sparkdl within Anaconda and using conda command? I am trying to run a project in ipython using pyspark and the code is using: from sparkdl import readImages everything work well except saprkdl which is not recognized and I could not find any way to install sparkdl for my Anaconda.

Your help is really appreciated Fari

fsahba avatar Aug 08 '17 18:08 fsahba

@fsahba making a package for pypi and conda is not easy because the python code calls all sorts of java libraries, which are also embedded. You need to start Spark and IPython using the spark shell (see [1] for example). See this issue on how you can then use the package: https://github.com/databricks/spark-deep-learning/issues/19

[1] https://stackoverflow.com/questions/31862293/how-to-load-ipython-shell-with-pyspark

thunterdb avatar Aug 10 '17 00:08 thunterdb