kellen

IO Windowing

2

Adds windowing filename support to file-based IOs. Additionally, makes file-based IOs more consistent in their `saveAs*` APIs. Adds to most file-based IOs: * `shardNameTemplate: String`, pattern-based filename format used by...

Currently transformOverride supports 1-to-1 mapping of input to output but should be flatMappable.

Provide shorthand for setting GenericRecord string type

1

Ala these docs but accounting for nullable and a dotted key path https://spotify.github.io/scio/extras/Sort-Merge-Bucket.html#genericrecords

Have two tests in the same repository, one which uses: ``` val (_, args) = ContextAndArgs(cmdLineArgs) ``` and one which uses ``` ContextAndArgs.typed[MyPipelineOptions](cmdLineArgs) ``` where both tests share a parameter...

These are currently missing from e.g. typedParquet but not parquetAvro See also #2824

Add GCS temp location to bigquery example

1

to run successfully, https://spotify.github.io/scio//extras/Scio-REPL.html#bigquery-example requires setting templocatiom to a GCS path ``` sc.options.setTempLocation("gs://...") ```

At the top left of each code snippet, there are two links, "copy" and "source" that render as "copysource". "copy" is redundant to the copy icon at top right. "source"...

Add SparkeyIO

4

Adds `SparkeyIO`, makes sparkey endpoints testable in `JobTest`. See #3173 Open questions: * Should `SparkeyIO` be used on the write path and if so how? * Is there a better...

When writing a sparkey in one job via e.g. `asSparkey` and reading in another job via e.g. `sparkeySideInput`, testing the second job is clunky. Ideally a `SparkeyIO` class would be...