scio

scio copied to clipboard

Move scio-avro to magnolify-avro

2

Currently scio-avro uses `com.spotify.scio.avro.types.AvroType[T]` that implements "case class GenericRecord" codec using `scala.reflect.macros._` in `com.spotify.scio.avro.types.TypeProvider`. This was implemented in 2017-2019. - Move AvroIO to a newer implementation based on [magnolify-avro](https://github.com/spotify/magnolify/blob/main/docs/avro.md) using...

shnapz

When using `largeHashIntersectByKey`, we've experienced pipelines hanging when running with dataflow default runnuer. It seems that enabling the [runner v2](https://cloud.google.com/dataflow/docs/runner-v2) unblocks the situation, but we should check why sparkey blocks...

RustedBones

## About this PR 📦 Updates [org.tensorflow:tensorflow-core-api](https://www.tensorflow.org) from `0.4.2` to `0.5.0` ## Usage ✅ **Please merge!** I'll automatically update this PR to resolve conflicts as long as you don't change...

scala-steward

and add example

kellen

new StateBaseAsyncDoFn with State and Timers

6

Feature Request, motivated by BaseAsyncDoFn and KV lookups to avoid duplicates (instead of using Redis, BigTable, Hazelcast IMDG, etc) https://github.com/albertols/scio-db/blob/develop/src/main/scala/com/db/myproject/async/http/state/StateBaseAsyncDoFn.java 1. By using State and Timers we could prevent **processElement**...

albertols

Support projections in ParquetAvroFileOperations/ParquetAvroSortedBucketIO

1

ParquetAvroFileOperations always overrides the "projection" option to equal the full reflected schema, so you can't supply a projection for a SpecificRecord class: https://github.com/spotify/scio/blob/110f79593c67c58a2c2465bf2fb340ff4711003f/scio-smb/src/main/java/org/apache/beam/sdk/extensions/smb/ParquetAvroFileOperations.java#L175-L176

clairemcginty

Move to hbase bigtable

1

Missing hbase magnolify in the example

RustedBones

BigQuery IT test with null optional and empty repeated values

2

RustedBones

Add option/env var indicating job should be cancelled on interrupt

1

Capture in `waitUntilFinish` and add a shutdown hook

kellen

ParquetTypeSortedBucketIO writes and reads don't work with CaseMapper

1

key getter does not work when a CaseMapper is required to translate field names: ``` [info] Cause: java.lang.IllegalStateException: Leaf key field user_name does not exist in record class class org.apache.beam.sdk.extensions.smb.ParquetEndToEndTest$Event...

clairemcginty