scio
scio copied to clipboard
spotify
A Scala API for Apache Beam and Google Cloud Dataflow.
It's been brought up internally, basically [this](https://github.com/spotify/scio/blob/master/scio-avro/src/main/scala/com/spotify/scio/avro/AvroIO.scala#L89) but in the form of a `FileOperations`
We have some awesome transforms in Scio's [transforms package](https://github.com/spotify/scio/tree/master/scio-core/src/main/java/com/spotify/scio/transforms). When using them in Scala pipelines, it can be less idiomatic. It would be nice to have first class support for...
This issue happens on serde when Java `serialVersionUID` is checked. Scala 2.12.x doesn't extensively set `@SerialVersionUID` across collections which means that in order for this to work in Flink one...
`sc.distCache` can be used to download some files and distribute them across workers. However, currently it can be done only once when `DistCache.apply()` is called. I need to update local...
I have a pipeline that does something like this on Scio 0.9.2 ``` val a = sc.parquetAvroFile[MySchema](path, Projection) val b = a.filter(xyz) val c = a.map(a => abc) ``` and...
it seems most of the implementation of `ScioIO[T]` is either read-only or write-only. For some IOs, read types are not as same as write types, that reduce the value of...
Is there an example of FileIO.WriteDynamic for Parquet files. I see that there is saveAsDynamicTextFile and saveAsDynamicAvroFile. It would be great if we can have a similar saveAsDynamicParquetFile API.
Trying to write a `SCollection[T: Schema]` using `.saveAsBigQuery(..)` I get a NPE because Beam doesn't know the `TableSchema` This originates from [here](https://github.com/spotify/scio/blob/master/scio-bigquery/src/main/scala/com/spotify/scio/bigquery/BigQueryIO.scala#L730)
Here’s an incomplete list of tasks. We can break them down further or create new issues to track as we go. - [x] Make sure Flink runner runs all Beam...