ecosystem

ecosystem copied to clipboard

Does this connector work with TF 2.x?

4

The latest TF right now is 2.4.0. The latest connector on maven central is 1.15.0 published on Oct 23, 2019. If I build the connector with the instructions from here:...

dgoldenberg-audiomack

Failed to build spark-tensorflow-connector_2.12

3

**Environment** *** tensorflow: 2.0.0 spark version: 2.4.3-107 maven version: 3.6.3 *** **Description** I have followed the steps to build which is shown in the [link](https://github.com/tensorflow/ecosystem/tree/master/spark/spark-tensorflow-connector) .  But, I got...

Echo9573

The lib doesn't seem to be working in the context of TensorFlow 2.x. My environment: - AWS emr-5.31.0 - TF=2.1.0, Spark=3.0.1 - built the library with the following: ``` mvn...

dgoldenberg-audiomack

java.math.BigDecimal cannot be cast to org.apache.spark.sql.types.Decimal

1

Hi, I am trying to save a Spark dataframe using the connector. The dataframe schema is: root |-- CustomerNumber: string (nullable = true) |-- ProductId: string (nullable = true) |--...

mlarionov

I have the same problem as you. When I use a spark cluster with two or more nodes, the algorithm needs the same time as if I have only one...

DanyOrtu97

Distributed training is slow compared to single machine

2

I am trying to perform a distributed training on models from [Tensorflow Object Detection models](https://github.com/tensorflow/models/tree/master/research/object_detection). I am able to perform distributed learning, but to my notice the distributed training is...

saikishor

I currently have a local spark cluster 3.0 which consists of 3 machines. Two machines have 2 NVIDIA GPUS and One machine is the spark client master which has no...

ghost

Prepare spark-tensorflow-distributor 0.1.0 release

4

* Use official Spark 3.0.0 release. * Bump version number to 0.1.0.

mengxr

GZIP/ZLIB compression for TFRecord files.

11

I am not 100% sure how reading compressed input is implemented in TensorFlow, but supporting outputting compressed TFRecord files would be amazing, as TFRecord is a rather inefficient format in...

thesuperzapper

https://github.com/tensorflow/ecosystem/blob/791a42f427139f2cb7bbae8a93a9e666a38c2bcc/spark/spark-tensorflow-connector/src/main/scala/org/tensorflow/spark/datasources/tfrecords/DefaultSource.scala#L173-L215 The check `if (dir.exists())` in line 179 above may cause subsequent partition write failed if partitions are more than 2 in the test below. Because partitions are written in...

simonzhaoms