scio
scio copied to clipboard
IO Windowing
Adds windowing filename support to file-based IOs. Additionally, makes file-based IOs more consistent in their saveAs*
APIs.
Adds to most file-based IOs:
-
shardNameTemplate: String
, pattern-based filename format used byDefaultFilenamePolicy.fromStandardParameters
-
filenamePolicyCreator: FilenamePolicyCreator
, a Single-Abstract-Method-able class taking apath
and asuffix
(required by most IO implementations) and returning aFilenamePolicy
. -
tempDirectory: String
, an optional temp directory location. Was previously supported by some IOs, but not all.
Supporting a more consistent interface required moving some sinks (e.g. BytesSink
) from FileIO.Sink
to FileBasedSink
to be used via WriteFiles
.
Codecov Report
Merging #4461 (af44f28) into main (ca2a3e5) will increase coverage by
0.80%
. The diff coverage is97.03%
.
@@ Coverage Diff @@
## main #4461 +/- ##
==========================================
+ Coverage 60.19% 60.99% +0.80%
==========================================
Files 279 286 +7
Lines 10110 10323 +213
Branches 853 840 -13
==========================================
+ Hits 6086 6297 +211
- Misses 4024 4026 +2
Impacted Files | Coverage Δ | |
---|---|---|
...ify/scio/io/dynamic/syntax/SCollectionSyntax.scala | 89.79% <ø> (ø) |
|
.../com/spotify/scio/values/WindowedSCollection.scala | 75.00% <ø> (ø) |
|
.../main/scala/com/spotify/scio/parquet/package.scala | 73.33% <73.33%> (ø) |
|
...rquet/types/dynamic/syntax/SCollectionSyntax.scala | 87.50% <87.50%> (ø) |
|
.../tensorflow/dynamic/syntax/SCollectionSyntax.scala | 88.88% <88.88%> (ø) |
|
...in/scala/com/spotify/scio/testing/ScioIOSpec.scala | 86.79% <88.88%> (+0.27%) |
:arrow_up: |
...arquet/avro/dynamic/syntax/SCollectionSyntax.scala | 92.30% <92.30%> (ø) |
|
.../src/main/scala/com/spotify/scio/io/BinaryIO.scala | 83.01% <94.59%> (+5.51%) |
:arrow_up: |
.../src/main/scala/com/spotify/scio/avro/AvroIO.scala | 92.85% <100.00%> (+2.96%) |
:arrow_up: |
...m/spotify/scio/avro/syntax/SCollectionSyntax.scala | 100.00% <100.00%> (ø) |
|
... and 22 more |
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.
Ready for re-review and/or further comments @clairemcginty @RustedBones