flatcar
Cgroup changes in Kernel 6.12 affects Java applications
Description
With Kernel 6.12 (currently part of beta release 4372.1.0) the behavior of cgroups changed and causes JVMs to not recognize cgroups correctly anymore.
The JVM fails to detected the container memory limits and takes the host memory as limit. If JVM are configured with -XX:MaxRAMPercentage=50, it will use 50% of the host memory, instead of 50% of the memory limit. This causes way higher heap usage and often results in OOMKill.
This is fixed in OpenJDK 25 and will very likely be backported to older versions. But this would mean, that all Java applications have to be rebuilt to work properly with Flatcar releases that contain Kernel 6.12.
Bug ticket for OpenJDK: https://bugs.openjdk.org/browse/JDK-8346874
There might be a possible workaround by building the kernel with CONFIG_CPUSETS_V1=y to restore the old cgroup behavior (or at least the part that is relevant for JVM to detect cgroups): https://bugzilla.redhat.com/show_bug.cgi?id=2334161
While this is not strictly speaking a bug of Flatcar itself, it might affect a lot of people and it should be discussed if there is a workaround on flatcar side.
Impact
Depending on the configuration, Java applications use more memory than the memory limit of a container and get OOMKilled.
Environment and steps to reproduce
Fetch memory settings from a java process:
k exec -ti java-pod -- java -XX:MaxRAMPercentage=50 -XshowSettings:vm --version
Picked up JAVA_TOOL_OPTIONS:
VM settings:
Max. Heap Size (Estimated): 31.34G
Using VM: OpenJDK 64-Bit Server VM
openjdk 21.0.8 2025-07-15 LTS
OpenJDK Runtime Environment Temurin-21.0.8+9 (build 21.0.8+9-LTS)
OpenJDK 64-Bit Server VM Temurin-21.0.8+9 (build 21.0.8+9-LTS, mixed mode, sharing)
In this case, the max heap size is 31Gi, which is half of the nodes memory. But the pod has a memory limit of 4Gi, so the max heap size should be 2Gi. The current stable release 4230.2.2 with kernel 6.6 detects it properly.
Expected behavior
Java applications detect the correct memory limit specified by the container.
Additional information
Discussion in matrix: https://matrix.to/#/!SakMJmpDOgVMlGukUp:matrix.org/$P4-06Of5TE_u2BGgg3L19ZXanaWpcY7JqChs_d0NPi8?via=matrix.org
Hi @thechristschn thanks again for raising this and for being a beta user. This should be solved in the next round of release, may I ask you to try again when it will be released? I'll let you know when it will be ready.
Hello @thechristschn, can you try again with the current Beta release (4426.1.0) ? Thanks 🙏
core@localhost ~ $ zgrep CONFIG_CPUSETS_V1 /proc/config.gz
CONFIG_CPUSETS_V1=y
Hello @tormath1,
already on it. But unfortunately, it still doesn't work as expected with 4426.1.0:
k exec -ti test-pod -- java -XshowSettings:vm -Xlog:os+container=debug --version
Picked up JAVA_TOOL_OPTIONS:
[0.000s][debug][os,container] Detected optional cpuset controller entry in /proc/cgroups
[0.000s][debug][os,container] Detected optional pids controller entry in /proc/cgroups
[0.000s][debug][os,container] controller memory is not enabled
[0.000s][debug][os,container] One or more required controllers disabled at kernel level.
VM settings:
Max. Heap Size (Estimated): 15.66G
Using VM: OpenJDK 64-Bit Server VM
openjdk 21.0.8 2025-07-15 LTS
OpenJDK Runtime Environment Temurin-21.0.8+9 (build 21.0.8+9-LTS)
OpenJDK 64-Bit Server VM Temurin-21.0.8+9 (build 21.0.8+9-LTS, mixed mode, sharing)
As comparison with Flatcar 4344.1.1:
k exec -ti test-pod -- java -XshowSettings:vm -Xlog:os+container=debug --version
Picked up JAVA_TOOL_OPTIONS:
[0.000s][debug][os,container] Detected optional pids controller entry in /proc/cgroups
[0.000s][debug][os,container] controller cpuset is not enabled
[0.000s][debug][os,container] controller memory is not enabled
[0.001s][debug][os,container] One or more required controllers disabled at kernel level.
VM settings:
Max. Heap Size (Estimated): 29.50G
Using VM: OpenJDK 64-Bit Server VM
openjdk 21.0.8 2025-07-15 LTS
OpenJDK Runtime Environment Temurin-21.0.8+9 (build 21.0.8+9-LTS)
OpenJDK 64-Bit Server VM Temurin-21.0.8+9 (build 21.0.8+9-LTS, mixed mode, sharing)
So I would say, the cpuset controller is now there as expected, but at least OpenJDK 21 requires an additional memory controller. I'll have a closer look into it.
If I look at how Debian fixed it: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1108294
We probably have to set
CONFIG_MEMCG_V1=y
as well.
@thechristschn thanks for the quick feedback. I can indeed see the same behavior. That's really weird, because I tested it here: https://github.com/flatcar/scripts/pull/3263#issue-3405405579 and it was working fine. Did OpenJDK release something in the meantime?
@tormath1 I can't reproduce your test from your last PR. I couldn't tell which exact java version was running in your fedora image or what the debug output said, which could have given us a hint about what went different there.
I've create https://github.com/flatcar/scripts/pull/3289 to add the other missing parameter.