rust-rdkafka
rust-rdkafka copied to clipboard
Queue Memory not freed when connection is closed from the clientsite
I have a web service which starts between 1-6 consumers for a topic in separate threads and assign the partitions manually for a request.
What I can see when I have a large topic and consume messages and write them into a sse channel and the connection is closed by the client (the web request) then the memory of queue rust-rdkafka / librdkafka is not freed an stays on a high level. When I consuming messages from a small topic which reaches "end of partition" than the memory of the queue's is freed and decrease.
example: a -> bigTopic 60 partitions with 40K men / s b -> smallTopic 1 partition reaching eof
request to topic a memory goes up to roundabout 1.2Gb decrease after connection aborted to roundabout 800MB do a second and third request on topic b the memory decrease to roundabout 100MB
define it in cargo.toml with rdkafka = { version = "0.33.2", default-features = false, features = ["libz", "cmake-build"] } When I do a compile with out a cmake-build then I get this as on the big topic => Message consumption error: NotImplemented (Local: Not implemented)
If I do a musl static build, from the memory point of view everything is ok, the memory is completely freed. The throughput then is really bad not acceptable.