spark-deep-learning
spark-deep-learning copied to clipboard
databricks
Deep Learning Pipelines for Apache Spark
Run all Unitest take almost 30m. I guess we may optimize the time.
In transformer's fit function I have following code: ``` word2vec = Word2Vec(vectorSize=self.getEmbeddingSize(), minCount=1, inputCol=self.getInputCol(), outputCol="word_embedding") vectorsDf = word2vec.fit( dataset.select(f.split(self.getInputCol(), "\\s+").alias(self.getInputCol()))).getVectors() # vectorsDf._sc._jsc is None ``` When you run the run-tests.sh...
I am new to SparkDL and basically following available resources. One of the video shows (https://databricks.com/blog/2017/06/06/databricks-vision-simplify-large-scale-deep-learning.html) , a Spark DL model got converted into Keras model before registering as an...
I follow the instructions: download the project and use build/sbt assembly and then I execute the python/run-tests.sh, but it gives me the following info: List of assembly jars found, the...
Java API
Hi, I have started to use Spark MLlib for my machine learning needs using Java. Now, I am curious to explore Deep Learning with Mllib but I find no documentation...
TensorframeOnSpark support the real distribute trainning on tensorflow . Maybe it's a good idea to port TensorframeOnSpark to spark deep learning project.
Loving this library, however, restricting to image data seriously constrains usage of the package in DL production systems. I think extending the API and docs to cover these use cases...
I check the last PR in spark-deep-learning is [KerasImageFileEstimator](https://github.com/databricks/spark-deep-learning/pull/35), and when i review the code, i find it will collect all trainning data to driver and then broadcast to executors....
It seems that most of the existing utilities are all specialized at learning on image data. Is there any plan to support general deep learning against non-image data? To be...
Trying to do hands-on exercises given in the main github page. While doing Tensorflow transformer example (with TFImageTransformer), from sparkdl import readImages, TFImageTransformer from sparkdl.transformers import utils import tensorflow as...
