enrich icon indicating copy to clipboard operation
enrich copied to clipboard

Published 20 hours ago • verified publisher icon snowplow

pubsub: sink without order

Open oguzhanunlu opened this issue 1 year ago • 0 comments
trafficstars

Currently all enrich apps sink chunks in order but we don't need order for PubSub. We need order for Kinesis and Kafka.

https://github.com/snowplow/enrich/blob/8f9123026396266bd9802de255313069a9bbbfa5/modules/common-fs2/src/main/scala/com/snowplowanalytics/snowplow/enrich/common/fs2/Enrich.scala#L95-L104

We should conditionally use parEvalMapUnordered for PubSub so that resource usage (CPU and mem) and throughput can improve. It is a matter of benchmarking to see how much improvement it'll bring.

oguzhanunlu avatar May 17 '24 09:05 oguzhanunlu