scio
scio copied to clipboard
(WIP) (#5296) Support Parquet predicates/projections in tests
WIP of Parquet projection/predicate support in JobTest.
Alternately, we could provide custom assertions similar to CoderAssertions
, i.e.:
val record: AvroType = ???
record withPredicate(FilterApi.and(...)) withProjection(...schema...) should eq(...)
but I think that's overall harder for users to work with.
The downside of this approach is that I'll have to implement separately for ParquetAvroIO, ParquetTypeIO, ParquetExampleIO, and SmbIO (TypeIO/ExampleIO won't need projections support but they could support filtering).
any feedback on the different possible approaches here is welcome!