micronaut-test icon indicating copy to clipboard operation
micronaut-test copied to clipboard

@Micronauttest-annotated tests can hang indefinitely on Java 21

Open j1m-renwick opened this issue 9 months ago • 6 comments

Expected Behavior

@Micronauttest tests never hang indefinitely

Actual Behaviour

When running our tests via gradle on the ubuntu-latest github runner, the testing can occassionally stop and hang indefinitely. The hanging always occurs after the following line is printed:

[Test worker] INFO io.micronaut.context.DefaultApplicationContext$RuntimeConfiguredEnvironment - Established active environments: [test]

However, the test that the hang occurs on is not consistent at all and changes each run, and sometimes the test suite will complete without incident. I estimate that approximately one in every 5 runs results in an indefinite hang condition.

It seems worth mentioning that some of the tests use Testcontainer Mongo and Redis container, although as mentioned these are not always the tests that the hang occurs on.

Steps To Reproduce

So far I have only been able to reproduce this locally by creating a docker container that approximates the github runner which has, and running it on 1 CPU (to approximate the 2vCPU that the normal github runner runs on for private repos).

# create the container with 1 CPU on ubuntu-latest
docker run -it --privileged --cpus="1" --cpuset-cpus="0" -v /Users/bob/myProject:/mnt/project ubuntu:latest

# Inside the container:

apt-get update
apt-get install -y docker.io wget unzip tar

# Download Java 21 adopt distribution which we specify the runner to use
wget https://github.com/adoptium/temurin21-binaries/releases/download/jdk-21.0.3%2B9/OpenJDK21U-jdk_x64_linux_hotspot_21.0.3_9.tar.gz -O /tmp/OpenJDK21U-jdk_x64_linux_hotspot_21.0.3_9.tar.gz

# Create a directory for the JDK installation
mkdir -p /usr/lib/jvm

# Extract the downloaded tar.gz file
tar -xzf /tmp/OpenJDK21U-jdk_x64_linux_hotspot_21.0.3_9.tar.gz -C /usr/lib/jvm

# Set up environment variables to use this Java version
export JAVA_HOME=/usr/lib/jvm/jdk-21.0.3+9
export PATH=$JAVA_HOME/bin:$PATH

# Add environment variables to the profile to make them persistent
echo "export JAVA_HOME=/usr/lib/jvm/jdk-21.0.3+9" >> /etc/profile.d/jdk21.sh
echo "export PATH=\$JAVA_HOME/bin:\$PATH" >> /etc/profile.d/jdk21.sh

# Source the profile to apply changes immediately
source /etc/profile.d/jdk21.sh

# Verify the installation
java -version

# start docker daemon as a background process
nohup dockerd &

# Verify Docker daemon is running
docker ps

# go to project directory
cd /mnt/project

./gradlew clean build test -i

As mentioned, the last command usually needs to be run several times to get the hang condition..

I took a thread dump of the longest-lived java process when the hanging state occurred:

threaddump.txt

Environment Information

Mentioned in the reproducing steps above for the docker image: OS - ubuntu-latest( aka noble / 24.04) JVM - 21.0.3 adopt JDK

Example Application

No response

Version

4.3.1

j1m-renwick avatar May 22 '24 16:05 j1m-renwick