rust-rdkafka icon indicating copy to clipboard operation
rust-rdkafka copied to clipboard

Queue Memory not freed when connection is closed from the clientsite

Open Algorab opened this issue 1 year ago • 0 comments

I have a web service which starts between 1-6 consumers for a topic in separate threads and assign the partitions manually for a request.

What I can see when I have a large topic and consume messages and write them into a sse channel and the connection is closed by the client (the web request) then the memory of queue rust-rdkafka / librdkafka is not freed an stays on a high level. When I consuming messages from a small topic which reaches "end of partition" than the memory of the queue's is freed and decrease.

example: a -> bigTopic 60 partitions with 40K men / s b -> smallTopic 1 partition reaching eof

request to topic a memory goes up to roundabout 1.2Gb decrease after connection aborted to roundabout 800MB do a second and third request on topic b the memory decrease to roundabout 100MB

define it in cargo.toml with rdkafka = { version = "0.33.2", default-features = false, features = ["libz", "cmake-build"] } When I do a compile with out a cmake-build then I get this as on the big topic => Message consumption error: NotImplemented (Local: Not implemented)

If I do a musl static build, from the memory point of view everything is ok, the memory is completely freed. The throughput then is really bad not acceptable.

Algorab avatar Aug 24 '23 13:08 Algorab