ydb-go-sdk icon indicating copy to clipboard operation
ydb-go-sdk copied to clipboard

feat: doesn't beaks on big batches to write

Open rekby opened this issue 1 year ago • 3 comments

Now the topic.Write will be failed if try to send more messages, than internal buffer has limit.

It isn't comfortable and it isn't purpose of the queue limit. Better choise:

  1. Acquire max limit and member overdraft https://github.com/ydb-platform/ydb-go-sdk/blob/1c9e3b5752a0014ace3a8803fdb3adfd113c2686/internal/topic/topicwriterinternal/writer_reconnector.go#L224-L224

  2. When release semaphore - first debt override. 2.1. Here (if weight more then zero) https://github.com/ydb-platform/ydb-go-sdk/blob/1c9e3b5752a0014ace3a8803fdb3adfd113c2686/internal/topic/topicwriterinternal/writer_reconnector.go#L241 2.2. Here https://github.com/ydb-platform/ydb-go-sdk/blob/1c9e3b5752a0014ace3a8803fdb3adfd113c2686/internal/topic/topicwriterinternal/writer_reconnector.go#L537

It will allow send big batches without errors and will protect from memory overflow on network/server problems.

rekby avatar Sep 25 '24 17:09 rekby