Lavalink
Lavalink copied to clipboard
lavalink-devs
RAM Leakage after continuous run
Description
The Lavalink Process Does not appear to clean up garbage it collects after some time after the start, I'm not sure why and it doesn't appear to be normal.
Version info
Client used: NodeJS Erela.JS v2.3.3
Output of java --version:
openjdk 17 2021-09-14
OpenJDK Runtime Environment (build 17+35-Ubuntu-120.04)
OpenJDK 64-Bit Server VM (build 17+35-Ubuntu-120.04, mixed mode, sharing)
Output of java -jar Lavalink.jar --version:
Version: 3.3.2.5
Build: 1222
Build time: 11.04.2021 08:07:27 UTC
Branch master
Commit: 7b914f2
Commit time: 09.04.2021 17:14:27 UTC
JVM: 17
Lavaplayer 1.3.76
After a Restart of Lavalink:
After ~7 Hours Runtime and 1 Player Playing:
62% free memory is not concerning at all. The garbage collector may decide that it doesn't need to clean
62% free memory is not concerning at all. The garbage collector may decide that it doesn't need to clean
I had ~30% RAM Usage from Lavalink after ~2 Days Runtime
You may want to limit your heap size using the -Xmx JVM flag. Say -Xmx2g for 2GB. 4GB is quite high and is enough for hundreds of players.
Lavalink doesn't appear to be leaking memory, it's just taking within the limit of what it is given
Sorry to bring this back to life. I notice the same issue (running lavalink on ptero). Using 1gb of ram and when it first starts it would use around 250mb than once my bot access it and plays a song it jumps to 512mb and will stay there. As each song is placed it adds another 30-60mb usage to the point it'll crash around 800/900mb (80-90% usage)
I'm using the latest dev build and haven't noticed any of this behaviour on a Debian bare metal server. However I have noticed that if I use a filter like low pass, it crashes if I allocate <1GB of RAM, it seemingly needs a LOT of memory for that filter in particular. If you have any filters enabled try allocating something like 2GB.
I have no filters. My startup flags are java -Xms10M -Xmx2048M -jar Lavalink.jar. On startup, I'm seeing 200mb usage than after a few songs played it gets up to 1gb! Doesn't seem to ever decrease the amount of ram while in use
That's just how Java works. It will use up to as much as you allocate it. It probably does go down you just don't really notice it. It only becomes a problem when you keep seeing "Killed" as that means either thr JVM is being memory starved or running out of memory either through lack of memory allocated or a leak.
So I did two test runs. For this, I actually used Sudhan's Discord-MusicBot. I also used Pterodactyl v1.7.0.
Run #1
I set my flags to java -Xms512M -Xmx512M -jar Lavalink.jar with the container having 2048m of ram.
- After startup ram usage was already at 716 MB (reported by ptero)
- Once loaded I played a song, stopped the song (via the bot), then started a new song. The ram usage jumped to 822MB
End result the ram usage kept increasing. Unfortunately, I forgot to get more screenshots of that test run.
Run #2
I set my flags to java -Xms1024M -Xmx1024M -jar Lavalink.jar with the container having 2048m of ram.
- After startup ram usage was already at 1.1 GB (reported by ptero)
- I played 2 songs in full then the music stopped playing around 30s into the 2nd song. Ram usage was at 1.38 GB
. No errors were reported in the console or bot.
- Each song added over 100MB of extra ram usage.
As end result, the music stopped playing though the player was still connected to the bot without errors.
My knowledge is very limited when it comes to programming and the understanding of Java. Primarily I come from the usage of Java in Minecraft. Yes, we give Java as much ram as possible but there is free ram and the server doesn't actually use all 7GB of ram unless it has to. Garbage collection helps reduce the amount of ram the server is using, mainly Aikar flags are used in this case).
Whether there's a memory leak or just poor configuration on my part and other people's there is something I'm missing here. The bot on average can only play around 2-5 songs before the music cuts out.
Try setting a lower xmx than 1024. Note that in the first run you set two minimums which looks like a typo. Expect total memory usage to be greater than the limit set.
If memory were getting constrained I would expect music to lag or the JVM to get killed, not for music to just stop.
Garbage collection helps reduce the amount of ram the server is using
Technically, but not really.
After startup ram usage was already at 1.1 GB (reported by ptero)
That's because you set xms to start it at 1 GB RAM
It could be that ptero is doing something weird because I have never seen this behavior.
So I tried with the same results doing 512m (max & min) with 1gb container.
The two xms was a typo in that message.
The music would lag sometimes but mainly it just start out cuts out. Starting a new song will result in nothing being played. Got to disconnect/reconnect the bot
Ya Ik I set the min to 1gb was just trying to cover everything from startup to end. Left a bit out but you get the picture.
With mc using aikar flags you have to remove 1gb from the flags for ptero overhead. Used that concept with this too and no luck.
Forgot to mention using Java 13 too as recommended in the readme. Whether or not that would affect anything idk. Personally would doubt it but that's my take.
Do you use any other flags?
Nope just the xms and xmx as mentioned. I have tried using java -Xms512M -Xmx512M -XX:+UseG1GC -XX:G1HeapRegionSize=4M -XX:+UnlockExperimentalVMOptions -XX:+ParallelRefProcEnabled -XX:+AlwaysPreTouch -jar Lavalink.jar which performed a bit better.
Whether there's a memory leak or just poor configuration on my part and other people's there is something I'm missing here.
It appears to just be you. This issue appears to have been opened because of a misunderstanding about how the JVM manages memory.
A socket in the CLOSE_WAIT state can actually cause this issue. Please check the issue below.
https://github.com/Walkyst/lavaplayer-fork/issues/86
This issue can actually be caused by the
CLOSE_WAITstate socket. Check out the issue below. Walkyst/lavaplayer-fork#86
That would make sense more than "misunderstanding about how the JVM manages memory". I'm running my lavalink inside of docker now and see have high memory issues