ydb-go-sdk icon indicating copy to clipboard operation
ydb-go-sdk copied to clipboard

feat: optimize topic message reader usage

Open pl553 opened this issue 1 year ago • 0 comments

Feature Request

Describe the Feature Request

When using the gzip codec, topic messages occupy a minimum size of more than 32 KiB, due to the gzip.Reader stored inside each message. The reader is not utilized until the message is actually read. This results in excessive memory usage and in our case led to OOM.

Describe Preferred Solution

The topic reader uncompresses messages in the background.

Describe Alternatives

The reader is created upon the first call to the Read method of the message.

Related Code

The reader for each message is created here: https://github.com/ydb-platform/ydb-go-sdk/blob/1b2f8d406425b28c18fae7301639c2a4204f5e70/internal/topic/topicreaderinternal/batch.go#L82

Message Read method: https://github.com/ydb-platform/ydb-go-sdk/blob/43b57d6c1463c492b99cb25d9655a66c7b36bd8f/internal/topic/topicreaderinternal/message.go#L72

Additional Context

If the feature request is approved, would you be willing to submit a PR? Yes

pl553 avatar Jul 16 '24 13:07 pl553