scio icon indicating copy to clipboard operation
scio copied to clipboard

(WIP) (#5296) Support Parquet predicates/projections in tests

Open clairemcginty opened this issue 3 months ago • 4 comments

WIP of Parquet projection/predicate support in JobTest.

Alternately, we could provide custom assertions similar to CoderAssertions, i.e.:

val record: AvroType = ???
record withPredicate(FilterApi.and(...)) withProjection(...schema...) should eq(...)

but I think that's overall harder for users to work with.

The downside of this approach is that I'll have to implement separately for ParquetAvroIO, ParquetTypeIO, ParquetExampleIO, and SmbIO (TypeIO/ExampleIO won't need projections support but they could support filtering).

any feedback on the different possible approaches here is welcome!

clairemcginty avatar Mar 19 '24 20:03 clairemcginty