scio
scio copied to clipboard
spotify
A Scala API for Apache Beam and Google Cloud Dataflow.
In the [documentation](https://spotify.github.io/scio/io/BigQuery.html#bigquerytype-fromquery) I can find an example of how to pass arguments to `@BigQueryType.fromQuery`: ``` // generate schema at compile time from a specific date @BigQueryType.fromQuery("SELECT user, url FROM...
When launching a replacement job for a streaming pipeline in dataflow, `transformNameMapping` option must be given when transformation names have changed. Scio by default use the callsite for the transformation...
It looks like newly added [covary* functions](https://github.com/spotify/scio/blob/main/scio-core/src/main/scala/com/spotify/scio/values/SCollection.scala#L331-L337) don't work when applied directly to the input `transform` `SCollection` e.g.: ```scala def main(cmdlineArgs: Array[String]): Unit = { val (sc, args) = ContextAndArgs(cmdlineArgs)...
These are currently missing from e.g. typedParquet but not parquetAvro See also #2824
I am following the [getting started](https://spotify.github.io/scio/Getting-Started.html) guide of scio. When I run: ``` runMain com.spotify.scio.examples.WordCount --project=my-gcp-project --runner=DataflowRunner --zone=us-west1 --input=gs://apache-beam-samples/shakespeare/kinglear.txt --output=gs://XXX/scio/wordcount ``` Two things happen: - first of all, it does...
There is a pipeline which has been consistently getting stuck on attempt to write to JDBC. The thread dump on one worker revealed a bunch of threads waiting for a...
initCounter works by creating a new root-node transform that initializes the counter value to 0. However, Dataflow runner now separates out counter values *per step*. So the 0 value from...
We have [RateLimiterDoFn](https://github.com/spotify/scio/blob/be7166c58450bd59ae5d0048d39ef3ea0d5ed107/scio-core/src/main/java/com/spotify/scio/transforms/RateLimiterDoFn.java) but it's a bit clunky for the average user to drop down to the java `DoFn` API directly. It could be nice to have an API like...