pulsar-beam icon indicating copy to clipboard operation
pulsar-beam copied to clipboard

These commit is mostly resolve the memory leak problem and optimize memory usage to handle millions of requests

Open truong-hua opened this issue 3 years ago • 0 comments

These commits optimize memory allocation to reuse buffer instead of keep allocating and release the memory on every request which push a lot of pressure on GC.

This pull request also add a notice and an config option to prevent memory leak if client sending Authorization while Pulsar broker does not activate it. And an option to keeps both Pulsar token authorization and normal HTTP Authorization without confliction.

This pull request also allow to set a limit on number of concurrency (config WorkerPoolSize) to limit number of concurrent request to protect the brokers as same as leverage reusable memory buffer. Without this one, an accidentally peak time can cause crashing of service and uncontrolable of resource usage.

This pull request is tested and using on production environment of https://doopage.com (we are using only Pulsar receiver) to handle few millions requests per day and it definitely improve the performance by using less CPU (mostly spend on GC) and of course much less Memory.

truong-hua avatar Jul 13 '22 07:07 truong-hua