Kenn Knowles
Kenn Knowles
@abacn @reuvenlax what are your thoughts on this change in options for the new API?
> Is the problem that the table is precreated but the schema passed into BigQueryIO contains fields not in the actual table? That's what this reads like to me, yes....
I think this is a fine idea.
I think the problem is that no runner actually implements RequiresStableInput. If Dataflow implemented it, the implemention would _be_ a shuffle.
> There is still a shuffle of the offsets due to the combine-per-key before the offset commit fn. If there was not that shuffle, having RequiresStableInput insert a shuffle would...
> This change causes an error when updating the Dataflow pipeline from previous version, but the update can be allowed by passing: > > ``` > --transformNameMapping={"KafkaIO.Read/KafkaIO.Read/KafkaIO.Read.ReadFromKafkaViaSDF/KafkaIO.ReadSourceDescriptors/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey":""}" > ``` >...
This could even impact jobs just running as cronjobs, right?
Pinging @je-ik and @thebozzcl explicitly here
To test this thoroughly, let us add some of the postcommits by touching trigger files. In #32648 you can see how I edited the JSON files (including some new ones)...
I opened jto/beam#236 with some more trigger files. The "PVR" trigger files stands for "Portable Validates Runner" that isn't as directly impacted. I think the non-portable ValidatesRunner tests should test...