apache
Memory problem
Description
Hi Team, I encountered a problem when sending a large message (130M). During the sending process, the memory usage rose to 400M and was not reclaimed after the method ended. Even after I manually triggered a garbage collection, the memory did not drop back to within 20M. When receiving the message, the memory suddenly spiked to 1G. After I manually triggered a garbage collection, it dropped back to 900M.
Reproduction Steps
1.Send a big message（170M）： byte[] byteArray = null; using (FileStream fs = new FileStream(filePath, FileMode.Open, FileAccess.Read)) { byteArray = new byte[fs.Length]; fs.Read(byteArray, 0, byteArray.Length); } var messageIdT = await producer.NewMessage().Send(byteArray); 2.The memory will increase by 390M after Send() 3.Receive message await foreach (var message in consumer.Messages()) { await consumer.Acknowledge(message); } 4.When this message is received, the memory usage will start to rise, reaching one gigabyte, and after completing this cycle, the memory is not reclaimed.
Expected behavior
Want memory to be reclaimed
Actual behavior
I observed that when DotPulsar sends large messages, it takes some time. During this period, the memory usage starts to rise, and even after the sending is completed, the memory is not reclaimed. When receiving messages, the memory usage also rises sharply. And because it's in a foreach loop, I'm not sure if I can manually reclaim the memory.
Regression?
No response
Known Workarounds
No response
Configuration
- .net Framework 4.6.2
- DotPulsar 3.6.0
- Windows 10
- Any CPU
Other information
I'm sorry. I don't know
Hi @howeWang97 What version of Pulsar are you using (broker side) and have you configured the broker to accept messages larger than the default 5 mb, since you are able to send so large messages?
Hi @howeWang97 What version of Pulsar are you using (broker side) and have you configured the broker to accept messages larger than the default 5 mb, since you are able to send so large messages? Hi, Sir I'm using Pulsar version 4.0.0. I have already configured the maxMessageSize and dispatcherMaxReadSizeBytes in broker.conf, and set nettyMaxFrameSizeBytes in bookkeeper.conf to 500M.
This might come down to poor LOH handling in .NET Framework 4.6.2. Can you try using .NET 9 and see if the memory is reclaimed over time?
This might come down to poor LOH handling in .NET Framework 4.6.2. Can you try using .NET 9 and see if the memory is reclaimed over time?
Sir, I'm upgrading my project to. Net Framework 4.8 Memory still not reclaimed. As much as I'd like to upgrade the project to. Net 9 but my manager wouldn't allow the overall project to be upgraded to. Net Core. I will try to use it myself. Net 9 See if you have memory problems
Hi Sir @blankensteiner , I have tried using.NET 9 and memory has not been reclaimed. What should I do if I want to reclaim the message variable in "for each (v a r message in consumer . Messages()){}" before the end of a loop? Does the message variable have a method similar to Dispose? Is it possible for me to free up memory occupied by message if I use Marshal