Server crash due to mass GC request

[Supermarcel10]

Version modpack Used

1.20.x-0.7.12

Environment

Multiplayer

Side Crashed

Server

New Worlds

Yes

Steps to Reproduce

There does not seem to be any clear steps to reproduce this issue.

Some things can be ruled out based on the different states of the server.

• The world is a brand new world, with barely 3 hours of actual runtime.
• The issue occurs with player counts of 1 or above.
• The issue occurs no matter the allocated memory to the container (tested with 8, 16, 32 GiB).

General things spotted:

• The server seems to be making outbound upload in a similar pattern as the CPU. This is calls of 90 bytes, and it happens immediately no matter if players are present.
• The server seems mostly stable, but sometimes goes into overdrive and utilises unexpectedly high CPU time.
  • There is no specific time when this occurs. It may occur both while exploring, or while staying mostly still.
• The server originally ever utilised 2.79 GiB while utilising the -XX:MaxRAMPercentage=95.0 argument.
  • Despite more memory being allocated, it would often drop by a few MiB. This would imply GC is being hit, cleaning up memory.
  • This is unusual since GC likes to only trigger once a higher threshold is met.

Java arguments that were tried on the server:

1. java -Xms128M -Xmx8192M -Dterminal.jline=false -Dterminal.ansi=true $( [[ ! -f unix_args.txt ]] && printf %s "-jar minecraft_server.jar" || printf %s "@unix_args.txt" )
2. java -Xms128M -XX:MaxRAMPercentage=95.0 -Dterminal.jline=false -Dterminal.ansi=true $( [[ ! -f unix_args.txt ]] && printf %s "-jar minecraft_server.jar" || printf %s "@unix_args.txt" )

Crash Report

crash-2024-09-07_00.37.58-server.txt debug-5.log

Additional Information

This is likely caused by a broken mod as part of the server modpack.

The container itself does not show any signs of nearing the allocated memory, but the CPU is showing significant red flags.

Server in a "semi-normal" state:

This would imply to me that GC is being called almost every 2 seconds or so, causing the server to kill itself by constantly calling GC for a non-existent problem with memory.

Server in a crashed state:

[Supermarcel10]

Taking a look further at this, it seems the server never exceeds 3076 MiB, even under synthetic workloads. Under synthetic workloads, the server then crashes in about 240 seconds of starting with the exception OutOfMemoryError: Out of heap space.

[Supermarcel10]

Information

I've found a solution to this issue. It seems the modpack uses an arbitrary launcher, which does not pass through the allocated memory to the server JVM.

I will leave this issue open, since it seems inherently unknown why the modpack abstracts the server behind a launcher for no apparent reason.

Notes

• I have manually updated to use forge version 1.20.1-47.3.0 as part of testing, however I believe it may not be required for the fix.
• My solution utilises containerisation, but the same principles should apply on bare-metal instances.
• My solution utilises Linux as the base OS, but the same principles should apply on a Windows based OS - just ensure you replace all unix_args.txt with win_args.txt.

Solution

In my container I copied over the run configuration $CONTAINER/libraries/net/minecraftforge/forge/1.20.1-47.3.0/unix_args.txt to $CONTAINER/unix_args.txt. Your version may vary if you have not updated forge, but same results should apply.

You can clean up any jar files in the root of the container, including deleting minecraft_server.jar. Prior to executing the java command change the parameters to printf %s "@unix_args.txt". You should then be able to launch the container like normal. It should start up a lot faster, because it's no longer executing via a separate launcher.

My exact jvm arguments are as follows:

java -Xms128M -XX:MaxRAMPercentage=98.0 -Dterminal.jline=false -Dterminal.ansi=true $( [[ ! -f unix_args.txt ]] && printf %s "-jar minecraft_server.jar" || printf %s "@unix_args.txt" )

These arguments provide me with the best flexibility working with execution via unix_arg injection and via direct jar execution on different container hardware configurations.