scio

scio copied to clipboard

Make the temporary table prefix and target dataset configurable in the MockBigQuery class

2

Actually temporary table prefix and staging dataset are defined in TableOps. This prevents us from specifying them when calling the mockTable method in the MockBigQuery. https://github.com/spotify/scio/blob/main/scio-google-cloud-platform/src/main/scala/com/spotify/scio/bigquery/client/TableOps.scala#L48 We need this because...

azaroui

Currently transformOverride supports 1-to-1 mapping of input to output but should be flatMappable.

kellen

Methods from saveAsBigQuery category don't expose crucial Beam BigQueryIO.writers parameters

3

I would like to discuss API of methods from saveAsBigQuery category. The methods provide very high level of abstraction for underlaying Beam API. In general this is expected behaviour to...

mkuthan

Provide shorthand for setting GenericRecord string type

1

Ala these docs but accounting for nullable and a dotted key path https://spotify.github.io/scio/extras/Sort-Merge-Bucket.html#genericrecords

kellen

Improve KV batch API

1

I realized that sio is only having an API for beam `GroupIntoBatches` with `ofSize` 1. this can be problematic in streaming pipelines when keys have few elements and batch takes...

RustedBones

Bumps [JamesIves/github-pages-deploy-action](https://github.com/JamesIves/github-pages-deploy-action) from 4.3.4 to 4.4.0. Release notes Sourced from JamesIves/github-pages-deploy-action's releases. v4.4.0 What's Changed Adding tag option to action by @​germa89 in JamesIves/github-pages-deploy-action#1142 - this can be used by...

dependabot[bot]

Have two tests in the same repository, one which uses: ``` val (_, args) = ContextAndArgs(cmdLineArgs) ``` and one which uses ``` ContextAndArgs.typed[MyPipelineOptions](cmdLineArgs) ``` where both tests share a parameter...

kellen

Added an optional header to saveAsDynamicTextFile, mostly for catering the CSV files! This would not be a breaking change, since default value of header is null, and if header is...

ruchirsachdeva

feat(pubsub-client): extended PubsubIO functionality to support custom Pubsub clients

1

Scio previously didn't offer a clean way for developers to create their own custom Pubsub clients. A standard workaround that I'd done in the past was vendoring Apache Beam internals...

chiragbharadwaj

Cleanup planned deprecation for `v0.13.0` milestone - [ ] elasticsearch 6 module (end of life)

RustedBones