snowplow-rdb-loader icon indicating copy to clipboard operation
snowplow-rdb-loader copied to clipboard

Transformer Kinesis: create micro batches according to their size

Open spenes opened this issue 3 years ago • 0 comments

At the moment, micro batches in the window are created according to their count. However, this approach would lead to memory problems with oversized events. Instead, these micro batches should be created according to the total size of events in the buffer.

spenes avatar Sep 21 '22 14:09 spenes