spring-boot
spring-boot copied to clipboard
paketo-buildpacks
Add Actuator client for Docker HEALTHCHECK
This is just a feature request, so not sure the default template applies.
Feature request
I'd like to set up a Docker HEALTHCHECK for a Spring Boot app running Spring Boot Actuator. Using the default Liberica JRE, curl isn't present, so something like the following fails:
healthcheck:
test: [ "CMD", "curl", "-f", "http://localhost:8080/health-check"]
interval: 5s
timeout: 2s
start_period: 15s
retries: 20
I thought "I guess I'll write a buildpack with some lightweight HTTP client in Java that I can reference from test:", as explained nicely in this blog post. And then thought "Wait, I'm already using a Spring Boot buildpack, is this a regular enough use case for it to be included with the Spring Boot buildpack itself?" 🙂
The present advice is to include code with your application that you can execute to trigger the HTTP request to your health check. You can then set the health check to java com.my.HealthCheck or whatever you need to run.
This use case has come up a couple of times though and the Paketo working group has talked about adding a buildpack to cover this case, which is specific to running with Docker. On K8s, you can use its built-in health check capabilities. I think we'd want this to be added in a generic way that would work across all buildpacks, so a small self-contained binary. We'd also talked about how this binary needs to have limited functionality so that it's just for local health checks. We don't include curl because it's a tool that can do a lot of things, which has security implications (not that curl is insecure, but it can be easily used by an adversary).
Anyway, if you're interested in contributing something, I'd be happy to chat more. Either here, or if you want to chat on the Paketo team Slack. https://paketo.io/community/
Thank you for the background and the suggestions @dmikusa-pivotal!
I think we'd want this to be added in a generic way that would work across all buildpacks, so a small self-contained binary.
I was originally thinking an Actuator-specific implementation would be good to be able to parse the /health JSON response. But given support for /actuator/health/readiness in 2.3, I see the benefit of a generic client. Using a small binary configured at build time to only hit that URL so it can't be changed at runtime sounds like a more robust approach.
Are there already thoughts about what that technologies would be preferred? A binary would need to compiled to native amd64 or arm64. The benefit of using something for the JVM is it shouldn't have issues with multi-arch images, but then it would only work for JVM apps.
Are there already thoughts about what that technologies would be preferred? A binary would need to compiled to native amd64 or arm64. The benefit of using something for the JVM is it shouldn't have issues with multi-arch images, but then it would only work for JVM apps.
Some requirements:
- It would have to be a self-contained binary. The binary should work on all of the stacks we support, so tiny, base & full.
- Right now we're only supporting amd64 builds, but hopefully, in the near future, we'll be supporting arm64 as well. Compilation for multiple targets would be a plus.
- Produce as small/compact of a binary as possible so that we are space consensus.
- Include automation. The Paketo team has to manage the tool & buildpack so we need to have automation in place. For Go, we have that already so we could probably reuse what we already have. For other languages, that would be something that would need to come with the PR.
Go would be suggested, and probably the easiest path. If you're interested in another language like Java native image or Rust, that might be possible too. Just let me know what you're thinking.
The present advice is to include code with your application that you can execute to trigger the HTTP request to your health check. You can then set the health check to
java com.my.HealthCheckor whatever you need to run.
Based on the quote above from @dmikusa-pivotal I should be able to run a java application that does the actual healthcheck.
This does not work if you have built your image with the gradle task bootBuildImage with default configuration.
That image does not hava the java executable in the PATH for the cnb user.
The error message from the healthcheck attempt shows this:
"Output": "OCI runtime exec failed: exec failed: container_linux.go:380: starting container process caused: exec: \"java\": executable file not found in $PATH: unknown"
Interesting @omattssonbam - Inside the container, the $PATH should be set & java should be on it. For example, if you were to use ProcessBuilder to run a sub-process, you could call java com.Foo and it should find java.
I didn't consider the point of view of the health check, which is outside the application environment. In this case, you need to invoke the launcher so it sets up the environment for the command.
Without the launcher:
> docker exec -it a9 java -version
OCI runtime exec failed: exec failed: container_linux.go:380: starting container process caused: exec: "java": executable file not found in $PATH: unknown
With the launcher:
> docker exec -it a9 launcher java -version
Setting Active Processor Count to 6
Calculating JVM memory based on 7258636K available memory
`For more information on this calculation, see https://paketo.io/docs/reference/java-reference/#memory-calculator
Calculated JVM Memory Configuration: -XX:MaxDirectMemorySize=10M -Xmx6862369K -XX:MaxMetaspaceSize=89066K -XX:ReservedCodeCacheSize=240M -Xss1M (Total Memory: 7258636K, Thread Count: 50, Loaded Class Count: 13311, Headroom: 0%)
Enabling Java Native Memory Tracking
Adding 128 container CA certificates to JVM truststore
Spring Cloud Bindings Enabled
Picked up JAVA_TOOL_OPTIONS: -Djava.security.properties=/layers/paketo-buildpacks_bellsoft-liberica/java-security-properties/java-security.properties -XX:+ExitOnOutOfMemoryError -XX:ActiveProcessorCount=6 -XX:MaxDirectMemorySize=10M -Xmx6862369K -XX:MaxMetaspaceSize=89066K -XX:ReservedCodeCacheSize=240M -Xss1M -XX:+UnlockDiagnosticVMOptions -XX:NativeMemoryTracking=summary -XX:+PrintNMTStatistics -Dorg.springframework.cloud.bindings.boot.enable=true
OpenJDK 64-Bit Server VM warning: Native Memory Tracking did not setup properly, using wrong launcher?
OpenJDK 64-Bit Server VM warning: PrintNMTStatistics is disabled, because native memory tracking is not enabled
openjdk version "11.0.14.1" 2022-02-08 LTS
OpenJDK Runtime Environment (build 11.0.14.1+1-LTS)
OpenJDK 64-Bit Server VM (build 11.0.14.1+1-LTS, mixed mode)
The problem there is that it's going to run all of the pre-initialization logic, like the memory calculator, as well. That's probably not desirable. You likely don't want the same JVM memory argument set for the health check that you do for your main app.
You're probably better off just putting in the full path to the JRE.
> docker exec -it a9 /layers/paketo-buildpacks_bellsoft-liberica/jre/bin/java -version
openjdk version "11.0.14.1" 2022-02-08 LTS
OpenJDK Runtime Environment (build 11.0.14.1+1-LTS)
OpenJDK 64-Bit Server VM (build 11.0.14.1+1-LTS, mixed mode)
In this way, you bypass all of the pre-initialization logic so nothing runs but exactly what you put in your command.
That path should be stable, in that it won't change from build to build or if a new version of the JRE is installed. Where it would change is if you switch JVM vendors, say change from Bellsoft to Azul. The path will change in that case.
Thanks @dmikusa-pivotal, I had not found any information on how to use the launcher. My current workaround is a shell script that finds the java executable and use it to run a simple java application. The health-check is just
HealthCheck:
Command: [ "CMD-SHELL", "bash", "/workspace/BOOT-INF/classes/healthcheck.sh" ]
It works but feel brittle
This page has a little info if you're curious. It's mostly focused on how you can invoke the launcher to do things like start your app or alternative commands in the container: https://buildpacks.io/docs/app-developer-guide/run-an-app/
Your script should be fine, but as I said, the path to the java binary should be fairly static. If you hard code it into the image, it won't really change unless you're switching up Java vendors. Either way, it should run your health checker.
FYI, I have created an RFC proposing to introduce a buildpack for adding health check processes. See https://github.com/paketo-buildpacks/rfcs/pull/183 for details.
