scio
scio copied to clipboard
spotify
A Scala API for Apache Beam and Google Cloud Dataflow.
Currently we have `scio-cassandra3` should we have a `scio-cassandra4`; There are some client API changes but overall seems that what we have can:tm: work
I am trying to write to a clustered table using `saveAsTypedBigQueryTable` and getting: ``` is configured to use TableDestinationCoderV2. Set withClustering() on BigQueryIO.write() and, if you provided a custom DynamicDestinations...
Beam attempts to stage the jar for the current session as it is on the classpath, but it has not been created at this point. We need to create it...
Currently if you provide a quoted arg like: `--foo="bar"`, when you access `args("foo")`, you will get a string like `"\"bar\""`. This works differently than other command line arg abstractions I...
As discussed on Slack, this replaces https://github.com/spotify/scio/pull/3872 and #3750, fixes #3754 and fixes #3710. All the macros related code is now separated in its own folder and lives under `scala-2`.
This PR proposes a solution to the To.safe blocker in Scala 3 on which @MaximeKjaer, @liufengyun and I worked. It took us some time to get familiar with the API...
Adds `SparkeyIO`, makes sparkey endpoints testable in `JobTest`. See #3173 Open questions: * Should `SparkeyIO` be used on the write path and if so how? * Is there a better...
I haven't looked into this yet, but I'm seeing these popping up from time to time. ``` java.io.EOFException: Can't collect further: memorySource depleted at Due to Exception while trying to...
This is not a common use case, and we worked around it, but wanted to flag it as an issue. When we set `ImpersonatedCredentials` in `BigtableOptions`, and pass this to...
