Jun Shi
Jun Shi
I have never tested ArrayType(ArrayType(DoubleType). I think FloatType should work. You can try to cast to Float first. Make sure you use SequenceExamples.
I don't understand your use case. Can you elaborate on how you plan to use Spark-TFRecord with Tensorflow-Java? Spark-TFRecord is designed as a Spark data source, i.e. it handles data...
I am curious why batching can not be done on user side? I don't see the benefit of doing it inside the converter. Assuming you will feed the examples to...
Which Spark operation does batching correspond to? GroupBy? Spark-TFRecord is implemented as a Spark data source (similar to Avro, Parquet, CSV), so it supports most data source options. I don't...
It's not clear to me how to implement the logic in a Spark data source which basically is a format converter. Contributions are welcome.
[This](https://github.com/linkedin/spark-tfrecord/pull/50) is a PR for Spark 3.2 + 2.12. I plan to merge it this weekend. Scala 2.13 is tricky, I will see how easy it is. We don't have...
v0.5.0 (Spark 3.2 + Scala 2.13) has been published. https://search.maven.org/artifact/com.linkedin.sparktfrecord/spark-tfrecord_2.13/0.5.0/jar Please let me know if it works for you.
try this: `option("codec","org.apache.hadoop.io.compress.GzipCodec")`
@byronyi Thanks for porting it over here. Does this work for TF 2?
In addition, to use verbs, should I compile the files, then set the LD_LIBRARY_PATH to the .so file. The following automatically will work with stock TF releases, say 1.15 or...