MetaCubeX
[Bug] out of memory
Verify steps
- [X] I have read the documentation and understand the meaning of all the configuration items I have written, rather than just piling up seemingly useful options or default values.
- [X] I have carefully reviewed the documentation and have not resolved the issue.
- [X] I have searched the Issue Tracker for the issue I want to raise and did not find it.
- [ ] I am a non-Chinese user.
- [X] I have tested with the latest Alpha branch version, and the issue still persists.
- [X] I have provided the server and client configuration files and processes that can reproduce the issue locally, rather than a sanitized complex client configuration file.
- [X] I provided the simplest configuration that can be used to reproduce the errors in my report, rather than relying on remote servers or piling on a lot of unnecessary configurations for reproduction.
- [X] I have provided complete logs, rather than just the parts I think are useful out of confidence in my own intelligence.
- [X] I have directly reproduced the error using the Mihomo command-line program, rather than using other tools or scripts.
Operating System
Linux
System Version
Linux RT-AX86U-3E90 4.1.52 #2 SMP PREEMPT Sat Dec 3 14:03:02 EST 2022 aarch64 ASUSWRT-Merlin
Mihomo Version
mihomo-linux-arm64-v1.19.1
Configuration File
# HTTP 代理端口
port: 7890
# SOCKS5 代理端口
socks-port: 7891
# Linux 和 macOS 的 redir 代理端口
redir-port: 7892
# 允许局域网的连接
allow-lan: true
# 规则模式：Rule（规则） / Global（全局代理）/ Direct（全局直连）
mode: Rule
# 设置日志输出级别 (默认级别：silent，即不输出任何内容，以避免因日志内容过大而导致程序内存溢出）。
# 5 个级别：silent / info / warning / error / debug。级别越高日志输出量越大，越倾向于调试，若需要请自行开启。
log-level: error
# Clash 的 RESTful API
external-controller: "0.0.0.0:9090"
experimental:
ignore-resolve-fail: true
# RESTful API 的口令
secret: ""
Description
it panics due to out of memory after running for almost one day
Reproduction Steps
- start the program
- wait for one day
- it panics due to out of memory
Logs
time="2025-01-07T19:34:19.621770568+08:00" level=info msg="Start initial configuration in progress"
time="2025-01-07T19:34:19.623758325+08:00" level=info msg="Geodata Loader mode: memconservative"
time="2025-01-07T19:34:19.623796306+08:00" level=info msg="Geosite Matcher implementation: succinct"
time="2025-01-07T19:34:19.679491388+08:00" level=info msg="Initial configuration complete, total time: 57ms"
fatal error: runtime: out of memory
runtime stack:
runtime.throw({0xfa4165?, 0x23418?})
runtime/panic.go:1067 +0x38 fp=0x400199fb30 sp=0x400199fb00 pc=0x82438
runtime.sysMapOS(0x4002000000, 0x400000)
runtime/mem_linux.go:168 +0x104 fp=0x400199fb70 sp=0x400199fb30 pc=0x239f4
runtime.sysMap(0x4002000000, 0x400000, 0x3a740?)
runtime/mem.go:155 +0x34 fp=0x400199fb90 sp=0x400199fb70 pc=0x23424
runtime.(*mheap).grow(0x1c13340, 0x4?)
runtime/mheap.go:1539 +0x278 fp=0x400199fc10 sp=0x400199fb90 pc=0x371d8
runtime.(*mheap).allocSpan(0x1c13340, 0x4, 0x1, 0x0)
runtime/mheap.go:1244 +0x164 fp=0x400199fcb0 sp=0x400199fc10 pc=0x366d4
runtime.(*mheap).allocManual(0x400199fdb8?, 0x9000000000?, 0xc0?)
runtime/mheap.go:988 +0x28 fp=0x400199fce0 sp=0x400199fcb0 pc=0x36218
runtime.stackalloc(0x8000)
runtime/stack.go:408 +0x164 fp=0x400199fd70 sp=0x400199fce0 pc=0x63b04
runtime.copystack(0x4000747c00, 0x8000)
runtime/stack.go:884 +0x118 fp=0x400199fe70 sp=0x400199fd70 pc=0x649b8
runtime.newstack()
runtime/stack.go:1126 +0x36c fp=0x400199ffb0 sp=0x400199fe70 pc=0x6500c
runtime.morestack()
runtime/asm_arm64.s:342 +0x70 fp=0x400199ffb0 sp=0x400199ffb0 pc=0x88860
........
this is the full log when it crashed, hope it will help to analyze the problem, thanks in advance
Check if the memory is exhausted. There is no good solution for memory exhaustion.
I doubt that it's because there is not enough memory, if so, how to explain that it starts up properly, and running for one day and then panics due to out of memory
After I reboot the router, the program lives longer(several days till now), whereas before the reboot, the program almost panics as soon as I started it. So it seems like it's because there is not enough memory in deed... I want to verify that further, can you show me the minimum memory requirement of the program? @Skyxim
After I reboot the router, the program lives longer(several days till now), whereas before the reboot, the program almost panics as soon as I started it. So it seems like it's because there is not enough memory in deed... I want to verify that further, can you show me the minimum memory requirement of the program? @Skyxim
Memory is related to your configuration. You may need to test how much is needed to start under your own configuration. When there is no connection, the minimum memory. The number of proxy nodes, rules, rule sets, and whether to use GEOIP, etc. These belong to static memory and must be used at startup.
Depending on the number of active connections, TCP/UDP will need a cache. In particular, the QUIC class requires more memory. This part of the memory is the main memory usage.
ps. In my case, I start with about 20MB, and when there are about 30-60 active connections, it takes up about 60-80MB of memory.
Thanks for the quick response! I collected the following infos:
this is the memory usage of the system when the program panics:
total used free shared buffers cached
Mem: 933968 708636 225332 109600 0 173148
-/+ buffers/cache: 535488 398480
Swap: 0 0 0
this is the memory usage when the program works fine:
total used free shared buffers cached
Mem: 933968 517864 416104 6956 0 64124
-/+ buffers/cache: 453740 480228
Swap: 0 0 0
this is the memory usage seen from top -m currently with about 300 connections:
Mem total:933968 anon:84308 map:33204 free:416628
slab:261404 buf:0 cache:64124 dirty:0 write:0
Swap total:0 free:0
PID^^^VSZ^VSZRW RSS (SHR) DIRTY (SHR) STACK COMMAND
2786 1240m 84892 50356 0 33068 0 132 mihomo-linux-arm64-v1.19.1 -d /jffs/clash/
the free memory is indeed smaller when the program panics, but it seems there is still enough memory... can you show me some pointer? Thanks! @Skyxim
The system you are using may reserve the minimum available memory to ensure stability; this is not a program issue
Guys, I got dozens of Xiaomi AX3000T (ubootmod) flashed to OpenWRT 23.05.5 with basically the same configurations but there could be some minor tweaks in domain rules here and there. It has the same number of packages and, the same Mihomo core but sometimes in rare cases, OpenWRT kills clash via OOM killer because it eats up the whole free RAM. This could be the same bug mentioned here.
@Skyxim
ps. In my case, I start with about 20MB, and when there are about 30-60 active connections, it takes up about 60-80MB of memory.
If initially 30 connections take up 29 mb of memory, then when opening 200 connections, memory consumption will increase to 60 mb. If all connections are closed immediately, reopening them will consume new memory, even though more than enough memory has already been allocated. If there are only 30 reopened connections, the memory will not return to 29 mb, but will eventually drop to around 52-54 megabytes. The additional 10 new connections will consume extra memory, even though enough has already been allocated for 150 connections. It is definitely the case that memory is not being reused or freed somewhere, indicating a memory leak.
