martin trieu
martin trieu
Change classes that explicitly inject a `MetricTrackingWindmillServerStub` to take in a Function. This will give flexibility in later refactoring as we apply different ways to fetch the data without MetricTrackingWindmillServerStub....
Removed flaky logic around waiting in tests. Removed thread.sleep and replaced with triggers via CountDownLatch R: @scwhittle @Abacn ------------------------ Thank you for your contribution! Follow this checklist to help us...
No need to submit or review will be thrown away once testing is complete. there will be smaller PRs with the changes. ------------------------ Thank you for your contribution! Follow this...
StreamingDataflowWorker is a rather large class For maintainability it would be helpful in the long run to encapsulate different pieces of the worker harness logic into their own class. ------------------------...
Add a DirectCommitWorkStream that has a self contained commit queue, and commit thread. Callers will call `queueCommit` on the stream, and and commits will be batched and sent to windmill...
Add computation state cache and config loaders that load the computation state cache move status pages out of StreamingDataflowWorker file R: @scwhittle ------------------------ Thank you for your contribution! Follow this...
Add work processing context R: @scwhittle ------------------------ Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] Mention the appropriate...
Add direct path code path we will be able to run direct path pipelines by passing in the option is `IsWindmillServiceDirectPathEnabled` - I added some new components (they are just...
Previously, `StreamingEngineClient` would start and close streams after getting new worker metadata synchronously (close 1 stream at time, open 1 stream at a time). Make this happen in parallel and...
includes https://github.com/apache/beam/pull/31148 StreamingDataflowWorker will eventually just be constructing the harness runtime (different implementations for appliance, dispatched, fanout etc) and the main method. R: @scwhittle ------------------------ Thank you for your contribution!...