weaviate
Handle new `Acks` & `OOM` messages during streaming
Core PRs for ref:
- https://github.com/weaviate/weaviate/pull/9781
- https://github.com/weaviate/weaviate/pull/9911
This PR introduces handling of two new messages from the server when stream: Acks & Oom.
The Acks message is emitted from the server once it has received the batch from the client and added it to its internal queue. The client uses this to stop over-sending across the write and stream buffer by waiting after sending a batch until those objects have been Ack-ed. Then it can send the next batch.
The Oom message may be emitted while a batch ingestion process is in progress causing the server to run out of available resource. The client handles this by stopping sending objects, waiting for the server shutdown (interpreted as meaning the cluster is being scaled up), and then resuming once complete. If the stream is not re-established before a 5 minute timeout is reached, then the batch exits with a WeaviateBatchFailedToReestablishStreamError exception.
Codecov Report
:x: Patch coverage is 20.25316% with 63 lines in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 86.28%. Comparing base (fd58638) to head (fb9c1e7).
:warning: Report is 10 commits behind head on main.
| Files with missing lines | Patch % | Lines |
|---|---|---|
| weaviate/collections/batch/base.py | 14.49% | 59 Missing :warning: |
| weaviate/exceptions.py | 40.00% | 3 Missing :warning: |
| weaviate/connect/v4.py | 66.66% | 1 Missing :warning: |
Additional details and impacted files
@@ Coverage Diff @@
## main #1884 +/- ##
==========================================
- Coverage 86.45% 86.28% -0.17%
==========================================
Files 274 274
Lines 19966 20015 +49
==========================================
+ Hits 17261 17270 +9
- Misses 2705 2745 +40
:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.
:rocket: New features to boost your workflow:
- :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
