venice
venice copied to clipboard
[da-vinc-client] DVC aggregates push status updates across partitions
Summary
This PR enables the DVC to write batching push status updates across all partitions on a node.
During batch pushes, one DVC instance will send two events to the push statuses updates to push status system stores: one event indicating the push has started in this DVC node; and another event indicating that the push has completed in all partitions assigned to this DVC node so far.
A config to control this behavior: write.batching.push.status - by default, it's disabled. Will update default value after enough testing.
The batching only applies to batch push part, it doesn't affect incremental pushes so far.
How was this PR tested?
Added integration tests.
Does this PR introduce any user-facing changes?
- [x] No. You can skip the rest of this section.
- [ ] Yes. Make sure to explain your proposed changes and call out the behavior change.