scwhittle comments

Results 87 comments of


                                            scwhittle

[Bug]: Side Input Singleton View throw error when impulse period is short: PCollection with more than one element accessed as a singleton view

The issue here is that Latest.globally() is a separate triggered global Combine than the global Combine contained in View.asSingleton. Thus if Latest triggers multiple times, the subsequent singleton combine may...

Add direct path code path

@Abacn not sure how to turn off the reminders but this doesn't require a look. It is being broken apart into smaller PRs which I am reviewing.

Add direct path code path

@m-trieu Should we close this one for now? We can reopen after rebasing on top of the broken off PRs

Export gcs options

@Naireen Friendly ping, is this still relevant?

Clear side input caches of dofninvoker when finishing bundle. Add microbenchmarks for map_fn with side inputs.

fixes #28776

Clear side input caches of dofninvoker when finishing bundle. Add microbenchmarks for map_fn with side inputs.

fixed lint issue, remaining test failures seem unrelated but they are rerunning anyway

Clear side input caches of dofninvoker when finishing bundle. Add microbenchmarks for map_fn with side inputs.

@robertwb @tvalentyn as you have context from before and had performance concerns. From the benchmark, clearing on finish_bundle doesn't appear to impact runtime. You were correct that doing it every...

Clear side input caches of dofninvoker when finishing bundle. Add microbenchmarks for map_fn with side inputs.

Without the sdk cache enabled (not on by default), removing the PerWindowInvoker cache mans that there is lots of FnApi traffic to fetch side inputs which contributed to latency in...

Clear side input caches of dofninvoker when finishing bundle. Add microbenchmarks for map_fn with side inputs.

It would be pretty easy to do so but it could still cause latency regressions on pipelines that have a global side input that is not refreshing. I was looking...

remove processing/scheduling logic from StreamingDataflowWorker

I think the examples failure is unrelated so don't bother investigating it.