sparkflow icon indicating copy to clipboard operation
sparkflow copied to clipboard

incompatibility with TF2.3 keras optimizers

Open ra312 opened this issue 3 years ago • 0 comments

When initializing SparkAsyncDL with optimizer='adam', we get the following exception Traceback (most recent call last): File "spark-train.py", line 170, in p = Pipeline(stages=[va, spark_model]).fit(train) File "/usr/local/spark/python/pyspark/ml/base.py", line 132, in fit return self._fit(dataset) File "/usr/local/spark/python/pyspark/ml/pipeline.py", line 109, in _fit model = stage.fit(dataset) File "/usr/local/spark/python/pyspark/ml/base.py", line 132, in fit return self._fit(dataset) File "/data/#user_folder#/sparkflow/sparkflow/tensorflow_async.py", line 278, in _fit tf_optimizer = build_optimizer(self.getTfOptimizer(), self.getTfLearningRate(), optimizer_options)

tensorflow.version = 2.3.0 sparkflow = 0.7.0 File "/data/#user_folder#/sparkflow/sparkflow/tensorflow_async.py", line 20, in build_optimizer 'adam': tf.train.AdamOptimizer, AttributeError: module 'tensorflow._api.v2.train' has no attribute 'AdamOptimizer'

ra312 avatar Aug 25 '20 04:08 ra312