scio
scio copied to clipboard
spotify
A Scala API for Apache Beam and Google Cloud Dataflow.
e.g. when using: ```scala val scoll: SCollection[MyMessage] = ??? scoll.write(PubsubIO.proto[MyMessage](topic))(PubsubIO.WriteParam()) ``` IntelliJ will infer `Nothing` as the type for the second argument to `write`. Not sure how/if this is possible...
Beam's new way of writing custom do function is to use annotations in the `@ProcessElement` function. Check that `ProcessContext` is only used where relevant Update to newer beam style when...
Updates [joda-time:joda-time](https://github.com/JodaOrg/joda-time) from 2.10.14 to 2.11.1. [GitHub Release Notes](https://github.com/JodaOrg/joda-time/releases/tag/v2.11.1) - [Version Diff](https://github.com/JodaOrg/joda-time/compare/v2.10.14...v2.11.1) I'll automatically update this PR to resolve conflicts as long as you don't change it yourself. If you'd...
add scalafix rules for breaking changes in 0.12.0: https://spotify.github.io/scio/migrations/v0.12.0-Migration-Guide.html
Since `Row.query` is a string, you have to call `Row.query.format(args(0))`, etc.
MockByteArraySparkeyReader was not working properly since two `Array[Byte]`s containing the same bytes produce a different hashCode every time they're deserialized -- thus, the `Map#get` operation was throwing a `NoSuchElementException` even...
When upgrading from Scio 0.13.4 to 0.14.0, I noticed an issue has appeared in IntelliJ. In the following code, which produced no errors or warnings in 0.13.4, IntelliJ now complaints...
**CONTEXT** We are trying to use [saveAsBigQueryTable](https://github.com/spotify/scio/blob/edb07ff5904e8f9116aa78eeaee7b9e1a0f36d4a/scio-google-cloud-platform/src/main/scala/com/spotify/scio/bigquery/syntax/SCollectionSyntax.scala#L60) to append data to a table where additional fields might be added to the schema. At the moment, we are unable to do...
A bigger number of java libraries require java 11+ now. It is probably time to wish Java 8 farewell for scio too.