scio icon indicating copy to clipboard operation
scio copied to clipboard

IO Windowing

Open kellen opened this issue 1 year ago • 2 comments

Adds windowing filename support to file-based IOs. Additionally, makes file-based IOs more consistent in their saveAs* APIs.

Adds to most file-based IOs:

  • shardNameTemplate: String, pattern-based filename format used by DefaultFilenamePolicy.fromStandardParameters
  • filenamePolicyCreator: FilenamePolicyCreator, a Single-Abstract-Method-able class taking a path and a suffix (required by most IO implementations) and returning a FilenamePolicy.
  • tempDirectory: String, an optional temp directory location. Was previously supported by some IOs, but not all.

Supporting a more consistent interface required moving some sinks (e.g. BytesSink) from FileIO.Sink to FileBasedSink to be used via WriteFiles.

kellen avatar Jul 14 '22 18:07 kellen

Codecov Report

Merging #4461 (af44f28) into main (ca2a3e5) will increase coverage by 0.80%. The diff coverage is 97.03%.

@@            Coverage Diff             @@
##             main    #4461      +/-   ##
==========================================
+ Coverage   60.19%   60.99%   +0.80%     
==========================================
  Files         279      286       +7     
  Lines       10110    10323     +213     
  Branches      853      840      -13     
==========================================
+ Hits         6086     6297     +211     
- Misses       4024     4026       +2     
Impacted Files Coverage Δ
...ify/scio/io/dynamic/syntax/SCollectionSyntax.scala 89.79% <ø> (ø)
.../com/spotify/scio/values/WindowedSCollection.scala 75.00% <ø> (ø)
.../main/scala/com/spotify/scio/parquet/package.scala 73.33% <73.33%> (ø)
...rquet/types/dynamic/syntax/SCollectionSyntax.scala 87.50% <87.50%> (ø)
.../tensorflow/dynamic/syntax/SCollectionSyntax.scala 88.88% <88.88%> (ø)
...in/scala/com/spotify/scio/testing/ScioIOSpec.scala 86.79% <88.88%> (+0.27%) :arrow_up:
...arquet/avro/dynamic/syntax/SCollectionSyntax.scala 92.30% <92.30%> (ø)
.../src/main/scala/com/spotify/scio/io/BinaryIO.scala 83.01% <94.59%> (+5.51%) :arrow_up:
.../src/main/scala/com/spotify/scio/avro/AvroIO.scala 92.85% <100.00%> (+2.96%) :arrow_up:
...m/spotify/scio/avro/syntax/SCollectionSyntax.scala 100.00% <100.00%> (ø)
... and 22 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

codecov[bot] avatar Aug 10 '22 00:08 codecov[bot]

Ready for re-review and/or further comments @clairemcginty @RustedBones

kellen avatar Aug 17 '22 13:08 kellen