Exec user process caused: no such file or directory when using custom JRE and distroless base image

[georgeVasiliu]

Hello everyone, here I came scratching my head with this current issue related to Jib (I presume).

Environment:

• Jib version: 3.3.0
• Build tool: Gradle 7.5
• OS: Linux

Description of the issue:

This is a strange behavior that might or might not be connected actually to the Jib plugin, but I am left without any more other things to try on my own to be honest.

As can be seen from the plugin configuration, the working directory contains another directory called 'minimal-jre' which is produced by another bash script prior to having the Jib gradle plugin actually being ran.

The bash script simply runs jdeps over the source code and uses jlink with that output to create a custom JRE and place it in the 'minimal-jre' folder:

jlink --strip-debug \
      --no-header-files \
      --no-man-pages \
      --compress=2 \
      --add-modules $(cat jre-deps.info) \
      --output $TARGET_FOLDER

The extraDirectories is supposed to copy the contents from inside that folder into the container at the path '/minimal-jre' so ideally I would have my JRE present in the container at that part:

	extraDirectories {
		paths {
			path {
				from = file('minimal-jre') // Folder which contains the custom JRE.
				into = '/minimal-jre'
			}
		}
		permissions = ['/minimal-jre/bin/java': '755']
	}

We are pulling the gcr.io/distroless/java-base-debian11 with a specific SHA256 to get a base image which contains everything necessary to run the JVM, without a JDK/JRE in itself:

	image = 'gcr.io/distroless/java-base-debian11@sha256:b20ec5e5d7ba95a030e800dd45e2d01df60ef01bf7c99c7c33d367c89a2a4b24'

On top of that image, we should have the 'minimal-jre' folder inserted by extraDirectories, so theoretically we have a distroless image now with a custom JRE inside, and that is where we customize the entrypoint to point to the correct java binary:

 ['/minimal-jre/bin/java', '-cp', '@/app/jib-classpath-file', '@/app/jib-main-class-file', "$JAVA_OPTS".toString()]

Expected behavior:

When running the container image created by Jib, it should be up and running, however, when deploying to a local Rancher k8s distribution (with containerd as runtime, nerdctl as binary) the following error shows up:

 exec /minimal-jre/bin/java: no such file or directory

When running the container image created by Jib inside an Azure Kubernetes cluster, the following error shows up:

standard_init_linux.go:228: exec user process caused: no such file or directory

Clearly, there is something wrong with the entrypoint, or the extraDirectories not working as I supposed they are working, as both deployment clusters seem to believe that the '/minimal-jre/bin/java' does not exist or are unable to find them.

However, extraDirectories does validate the input path when evaluated, so if the 'minimal-jre' didn't exist before the Jib plugin ran, you'd encounter a build error. No build error means that it's able to parse the local folder and put it in the container.

Next thing I looked into was creating a tarball out of the image and looking into it's blobs, it is clear that the folder is actually present: (?!)

Why would the runtime then complain regarding this? Have I missed something? The base image should contain all the binaries required for the JVM, but the error clearly seems to be related to the entrypoint and the java binary not being found.

Steps to reproduce:

1. Just create a minimalist Spring Boot application that prints hello world (can copy any project).
2. Run jdeps over the fat jar produced by Spring Boot and run the jlink command from above
3. Use same jib configuration and check if the image is running

jib-gradle-plugin Configuration:

jib {
	extraDirectories {
		paths {
			path {
				from = file('minimal-jre') // Folder which contains the custom JRE.
				into = '/minimal-jre'
			}
		}
		permissions = ['/minimal-jre/bin/java': '755']
	}
	from {

		image = 'gcr.io/distroless/java-base-debian11@sha256:b20ec5e5d7ba95a030e800dd45e2d01df60ef01bf7c99c7c33d367c89a2a4b24'

	}
	to {
		image = "$CI_REGISTRY/$CI_PROJECT_PATH:$CI_COMMIT_SHORT_SHA"

	}
	container {
		ports = ['8080']
		format = 'OCI'
		entrypoint = ['/minimal-jre/bin/java', '-cp', '@/app/jib-classpath-file', '@/app/jib-main-class-file', "$JAVA_OPTS".toString()]
	}
	pluginExtensions {
		pluginExtension {
			implementation = 'com.google.cloud.tools.jib.gradle.extension.springboot.JibSpringBootExtension'
		}
	}
}

[mpeddada1]

@georgeVasiliu To double check that I'm understanding this correctly, does the container image created by jib run successfully when you build to the docker daemon and call docker run {container_built_by_jib}? Curious about whether it runs fine locally but fails on kubernetes or returns the same error in both environments.

[georgeVasiliu]

@mpeddada1 it is now having the same error at both local and Kubernetes, will come back with a bigger update later, I've made some progress on it and think I've found the actual issue behind it, but I still have not yet been able to solve it (so I've dropped the custom JRE and just use a simple distroless java17 for the moment, but I'm still pursuing to fix this issue)

[alicejli]

@georgeVasiliu were you able to determine the issue? Any additional details you can share would be helpful. Thanks!

[georgeVasiliu]

@alicejli unfortunately I had to move on and didn't get more time allocated for that issue :( we just moved on and kept using some base distroless image that already contained a JRE. However, I would still like to pursue this issue further.

I will create this weekend a small reproducible gradle setup with spring boot and jib and a custom sh script to jdep/jlink the project, and attempt the same things as before (since it was such a long time ago, due to how many updates passed by, maybe the issue was fixed ? ). I will give more updates this weekend on the results.

Thanks for the reminder for it!!!

[georgeVasiliu]

@alicejli a temporary update:

• I have created a new public repository in which I put together a minimal reproducible example;
• you can use the README.MD there to clone it & build the image locally & run it locally;
• as a first observation, when using Rancher Desktop (latest available version used, namely v1.9.1) as the container runtime (with dockerd engine, not the nerd one) the issue seems to have disappeared (there is no complaint about any missing files anymore);
• next test will be in the weekend with a deployment of the image to AKS and running it there in a test cluster and make sure it also doesn't complain about missing files anymore;

Here is the link to the repository : https://github.com/georgeVasiliu/spring-gradle-jib-jlink

I will come with another update over the weekend to check if the image is also running fine in the AKS environment.