ydb-python-sdk icon indicating copy to clipboard operation
ydb-python-sdk copied to clipboard

Support data limit when reading a batch with TopicReaderSync

Open JasonRammoray opened this issue 8 months ago • 0 comments

Allow a client to control the amount of data it receives, when reading a batch through TopicReaderSync.

Pull request type

Please check the type of change your PR introduces:

  • [x] Bugfix
  • [ ] Feature
  • [ ] Code style update (formatting, renaming)
  • [ ] Refactoring (no functional changes, no api changes)
  • [ ] Build related changes
  • [ ] Documentation content changes
  • [ ] Other (please describe):

What is the current behavior?

TopicReaderSync.receive_batch ignores max_messages and max_bytes parameters, which means a client has no control over the amount of received data.

Issue Number: 365

What is the new behavior?

TopicReaderSync.receive_batch now takes max_messages and max_bytes into account.

Other information

Decisions made:

  1. Enforce at least one message and at least a single byte on a batch.
  2. Since a batch could theoretically be empty, don't use _commit_get_partition_session, and rather copy _partition_session from the batch to a new (sliced) batch.
  3. Make no slice in case if neither max_messages, nor max_bytes were provided.

JasonRammoray avatar Jun 09 '24 21:06 JasonRammoray