reuvenlax

I thought that the main bug had already been fixed.

We do cache the client instances. The problem is that today, each destination table requires a separate client. On Thu, Aug 25, 2022 at 8:45 AM Lukasz Cwik ***@***.***> wrote:...

Yes, but at least today there is no way of doing that AFAICT. We might have to wait for the BigQuery support to land. On Thu, Aug 25, 2022 at...

We are reusing clients across bundles for streaming write API. The DoFns that use this keep a global static cache of table name -> client On Fri, Aug 26, 2022...

Dataflow keeps track of the old and new schema, and will match up fields to ensure that the encoding position remains the same even if the field orders are different....

Sorting doesn't help here as a common use case is to add a new field on update. Dataflow at least should be able to handle schema update, as long as...

This is a risky change - are we sure that there are no customers relying on the old behavior?

If I understand this correctly, we are now propagating both ElementT and StorageApiWritePayload - correct? Doesn't this double the amount of data being processed?

I'm also wanting to know whether there was something motivating this change - i.e. is their a Beam user that currently needs this? In addition to being careful about perf,...