open-im-server icon indicating copy to clipboard operation
open-im-server copied to clipboard

Bug: memory leaking in MSGGATEWAY

Open Sylariam opened this issue 10 months ago • 2 comments

What happened?

This monday I upgraded server side to v3.6.0 using Helm Chart. For some reason MSGGATEWAY memory usage suddenly spiked, with no signs of stopping. This directly led to our IM service becoming unavailable, which coincided with our peak business hours. The quantity of error logs also increased during that period, including entries such as "read tcp :80->:5644: i/o timeout" and "websocket: close 1006 (abnormal closure): unexpected EOF".

MSGGATEWAY memory increased all the way to 11GiB. No stopping before our manual intervention. img_v3_029q_7c008829-eb5a-43d8-8146-90a9fc4b128g

OpenIM's official Grafana dashboard stats: img_v3_029q_4972ff6b-d852-47dc-b3c6-edb8a0d6c76g

Online users: image

What did you expect to happen?

no memory leaking plz

How can we reproduce it (as minimally and precisely as possible)?

Helm charts deploy v3.6.0

Anything else we need to know?

No response

version

```console $ {name} version # paste output here ```

Cloud provider

OS version

```console # On Linux: $ cat /etc/os-release # paste output here $ uname -a # paste output here # On Windows: C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture # paste output here ```

Install tools

Sylariam avatar Apr 10 '24 02:04 Sylariam

Setting aside the issue at hand, the current Kubernetes deployment lacks resource limitations, which would lead to unlimited consumption of cluster resources if any similar problems arise. This needs to be addressed.

Sylariam avatar Apr 10 '24 03:04 Sylariam

i got this problem too.

pokid avatar Apr 15 '24 01:04 pokid