Add dev services for kubernetes

[scrocquesel]

Closes #28189

Some design advices are more than welcome to adjust some behavior of dev service for kubernetes:

• Currently some integration tests like openshift-client use k8s mock server provided by AbstractKubernetesTestResource. This set systemProps to be read by fabric8 client and return config properties with key of the same key name. The kubernetes-client extension override those values with quarkus.* properties if any. The issue is that Dev services return the configuration as quarkus properties and if not disabled, it will override the configuration to connect to the Mock Server. Disabling dev services when mock server is used seems not doable. I guess a lot of people are using the Mock Server do to their tests and would experience a breaking change.

  Should dev service be disabled by default ? Should AbstractKubernetesTestResource be updated to return ``quarkus.` prefixed properties instead of fabric8 systemProps properties (not sure if this would work) ? Should we disable dev services manually in those project assuming the breaking change in behavior ?

• dev services for Kubernetes do its best to detect if a kubernetes configuration is defined so that the developer or test runner can use its own current kubeconfig. While, I think this is a valid default use case, it is not so easy to force dev services to start a cluster. This differs from how other dev services work because there is generally no such thing as a config file in home directory or env var for client of databases/message broker/..., and it should be rare a user defines master-url manually to disable dev services. Quarkus extension first build a fabric8 Config object and call autoconfigure on it inconditionally. If a kubeconfig is found, master-url will be set. Therefore, user needs to pass -Dkubernetes.auth.tryKubeConfig=false when running test or dev mode to bypass any kubeconfig present in home directory. Dev services use same logic to guess if master-url would be set at runtime and decide if a cluster should be started. In my case, and I think some user will be in the same case, I have a valid k8s cluster but still would prefer to use dev service cluster to isolate dev and tests and have automatic cleanup when stopping dev mode. Should a quarkus.kubernetes-client.auto-configure (or quarkus.kubernetes-client.devservices.override-kubeconfig`) be added to disable the fabric8 autoconfigure behavior ? At least, this property could be set in properties or dotenv file.

• Regarding docker image name, current testcontainer implementation used to start a api server only cluster do not allow to pass a docker image name for the api server and etcd containers. It is possible to specify a version if it is supported by the package (see https://github.com/dajudge/kindcontainer/blob/master/k8s-versions.json). If a user needs to change the registry name/path, it must provide a testcontainer image name substitutor. Do you have any concern about this ?

[quarkus-bot[bot]]

Thanks for your pull request!

The title of your pull request does not follow our editorial rules. Could you have a look?

• title should preferably start with an uppercase character (if it makes sense!)
• title should not start with chore/docs/feat/fix/refactor but be a proper sentence

This message is automatically generated by a bot.

[scrocquesel]

1/, I find out how to disable dev services when quarkus-test-kubernetes-client is imported. For now, it is one or another. 2/, added a devservices.override-kubeconfig property.

[quarkus-bot[bot]]

---

:waning_crescent_moon: This workflow status is outdated as a new workflow run has been triggered.

---

Failing Jobs - Building 6f5ce87978d6d8a47b0b9f3da1170a4faba98657

Status	Name	Step	Failures	Logs	Raw logs
:heavy_check_mark:	Gradle Tests - JDK 11
✖	Gradle Tests - JDK 11 Windows	Build	Failures	Logs	Raw logs
:heavy_check_mark:	JVM Tests - JDK 11
✖	JVM Tests - JDK 11 Windows	Build	Failures	Logs	Raw logs
:heavy_check_mark:	JVM Tests - JDK 17
✖	JVM Tests - JDK 17 MacOS M1	Set up runner	:warning: Check →	Logs	Raw logs
:heavy_check_mark:	JVM Tests - JDK 18

Full information is available in the Build summary check run.

Failures

:gear: Gradle Tests - JDK 11 Windows #

- Failing: integration-tests/gradle 

:package: integration-tests/gradle

✖ io.quarkus.gradle.BasicJavaLibraryModuleTest.testBasicMultiModuleBuild line 25 - More details - Source on GitHub

java.lang.AssertionError: 

Expecting path:

✖ io.quarkus.gradle.devmode.BasicJavaLibraryModuleDevModeTest.main line 23 - More details - Source on GitHub

org.awaitility.core.ConditionTimeoutException: Condition with lambda expression in io.quarkus.test.devmode.util.DevModeTestUtils that uses java.util.function.Supplier, java.util.function.Supplierjava.util.concurrent.atomic.AtomicReference was not fulfilled within 3 minutes.
	at org.awaitility.core.ConditionAwaiter.await(ConditionAwaiter.java:167)
	at org.awaitility.core.CallableCondition.await(CallableCondition.java:78)

:package: integration-tests/gradle/target/classes/basic-java-library-module/application

✖ org.acme.ApplicationConfigResourceTest.testAppConfigEndpoint() - More details - Source on GitHub

java.lang.RuntimeException: java.lang.RuntimeException: io.quarkus.builder.BuildException: Build failure: Build failed due to errors
	[error]: Build step io.quarkus.kubernetes.client.deployment.DevServicesKubernetesProcessor#setupKubernetesDevService threw an exception: java.lang.RuntimeException: java.lang.IllegalStateException: Previous attempts to find a Docker environment failed. Will not retry. Please see logs and check configuration
	at io.quarkus.kubernetes.client.deployment.DevServicesKubernetesProcessor.setupKubernetesDevService(DevServicesKubernetesProcessor.java:117)

---

:gear: JVM Tests - JDK 11 Windows #

- Failing: integration-tests/micrometer-prometheus 

:package: integration-tests/micrometer-prometheus

✖ io.quarkus.it.micrometer.prometheus.ClientRequestTest.testClientRequests - More details - Source on GitHub

java.lang.RuntimeException: 
java.lang.RuntimeException: io.quarkus.builder.BuildException: Build failure: Build failed due to errors
	[error]: Build step io.quarkus.kubernetes.client.deployment.DevServicesKubernetesProcessor#setupKubernetesDevService threw an exception: java.lang.RuntimeException: java.lang.IllegalStateException: Previous attempts to find a Docker environment failed. Will not retry. Please see logs and check configuration

[geoand]

Thanks for this!

We'll have to take a close look and get back to you.

[scrocquesel]

One last blocking issue, I can't find out how to not run integration test for JDK 11 Windows Container job. It fails because docker is not available in those jobs. Help wanted.

[quarkus-bot[bot]]

---

:waning_crescent_moon: This workflow status is outdated as a new workflow run has been triggered.

---

Failing Jobs - Building 8842e8ff6d229570486ec2813ec37630d04bbdce

Status	Name	Step	Failures	Logs	Raw logs
:heavy_check_mark:	Gradle Tests - JDK 11
✖	Gradle Tests - JDK 11 Windows	Build	Failures	Logs	Raw logs
:heavy_check_mark:	JVM Tests - JDK 11
✖	JVM Tests - JDK 17	Build	Failures	Logs	Raw logs
:heavy_check_mark:	JVM Tests - JDK 17 MacOS M1
:heavy_check_mark:	JVM Tests - JDK 18
:heavy_check_mark:	Maven Tests - JDK 11
:hourglass:	Maven Tests - JDK 11 Windows	Build	:warning: Check →	Logs	Raw logs
✖	Native Tests - Data2	Build	Failures	Logs	Raw logs

Full information is available in the Build summary check run.

Failures

:gear: Gradle Tests - JDK 11 Windows #

- Failing: integration-tests/gradle 

:package: integration-tests/gradle

✖ io.quarkus.gradle.BasicJavaLibraryModuleTest.testBasicMultiModuleBuild line 25 - More details - Source on GitHub

java.lang.AssertionError: 

Expecting path:

✖ io.quarkus.gradle.devmode.BasicJavaLibraryModuleDevModeTest.main line 23 - More details - Source on GitHub

org.awaitility.core.ConditionTimeoutException: Condition with lambda expression in io.quarkus.test.devmode.util.DevModeTestUtils that uses java.util.function.Supplier, java.util.function.Supplierjava.util.concurrent.atomic.AtomicReference was not fulfilled within 3 minutes.
	at org.awaitility.core.ConditionAwaiter.await(ConditionAwaiter.java:167)
	at org.awaitility.core.CallableCondition.await(CallableCondition.java:78)

:package: integration-tests/gradle/target/classes/basic-java-library-module/application

✖ org.acme.ApplicationConfigResourceTest.testAppConfigEndpoint() - More details - Source on GitHub

java.lang.RuntimeException: java.lang.RuntimeException: io.quarkus.builder.BuildException: Build failure: Build failed due to errors
	[error]: Build step io.quarkus.kubernetes.client.deployment.DevServicesKubernetesProcessor#setupKubernetesDevService threw an exception: java.lang.RuntimeException: java.lang.IllegalStateException: Previous attempts to find a Docker environment failed. Will not retry. Please see logs and check configuration
	at io.quarkus.kubernetes.client.deployment.DevServicesKubernetesProcessor.setupKubernetesDevService(DevServicesKubernetesProcessor.java:119)

---

:gear: JVM Tests - JDK 17 #

- Failing: extensions/flyway/deployment 
! Skipped: integration-tests/flyway integration-tests/hibernate-orm-tenancy/datasource integration-tests/hibernate-orm-tenancy/schema and 2 more

:package: extensions/flyway/deployment

✖ io.quarkus.flyway.test.FlywayExtensionInitSqlTest. - More details - Source on GitHub

java.lang.RuntimeException: java.lang.RuntimeException: Failed to start quarkus
	at io.quarkus.test.QuarkusUnitTest.beforeAll(QuarkusUnitTest.java:689)
	at org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.lambda$invokeBeforeAllCallbacks$12(ClassBasedTestDescriptor.java:395)

---

:gear: Native Tests - Data2 #

- Failing: integration-tests/jpa-db2 

:package: integration-tests/jpa-db2

✖ Failed to execute goal io.quarkus:quarkus-maven-plugin:999-SNAPSHOT:build (default) on project quarkus-integration-test-jpa-db2: Failed to build quarkus application

[iocanel]

One last blocking issue, I can't find out how to not run integration test for JDK 11 Windows Container job. It fails because docker is not available in those jobs. Help wanted.

@geoand? Any ideas?

[geoand]

I'll have a look later on

[geoand]

One last blocking issue, I can't find out how to not run integration test for JDK 11 Windows Container job. It fails because docker is not available in those jobs. Help wanted.

Not sure what you mean here. We intentionally don't run container related tests on Windows. Is there a test that is running that shouldn't be?

[scrocquesel]

One last blocking issue, I can't find out how to not run integration test for JDK 11 Windows Container job. It fails because docker is not available in those jobs. Help wanted.

Not sure what you mean here. We intentionally don't run container related tests on Windows. Is there a test that is running that shouldn't be?

That's what I observed on CI but not locally see https://github.com/scrocquesel/quarkus/actions/runs/3256150273

[scrocquesel]

One last blocking issue, I can't find out how to not run integration test for JDK 11 Windows Container job. It fails because docker is not available in those jobs. Help wanted.

Not sure what you mean here. We intentionally don't run container related tests on Windows. Is there a test that is running that shouldn't be?

That's what I observed on CI but not locally see https://github.com/scrocquesel/quarkus/actions/runs/3256150273

The test that should not run are those added by this PR

[geoand]

I still don't see which Kubernetes tests you say are failing on the Windows run.

[scrocquesel]

I still don't see which Kubernetes tests you say are failing on the Windows run.

You are right. As the job keep failing, I thought it was the new test.

[geoand]

Can you please rebase onto main so we can be sure nothing has broken in the meantime? If CI is good after that, let's merge

[quarkus-bot[bot]]

---

:waning_crescent_moon: This workflow status is outdated as a new workflow run has been triggered.

---

Failing Jobs - Building 524eecc331d5a48f9766e1ad85bdb1172dd302a5

Status	Name	Step	Failures	Logs	Raw logs
✖	Initial JDK 11 Build	Build	Failures	Logs	Raw logs

Failures

:gear: Initial JDK 11 Build #

- Failing: docs 

:package: docs

✖ Failed to execute goal org.asciidoctor:asciidoctor-maven-plugin:2.0.0:process-asciidoc (output-html) on project quarkus-documentation: Found 1 issue(s) of severity WARN or higher during conversion

[quarkus-bot[bot]]

Failing Jobs - Building 4f325803065262874c714e786e2394e904b048bc

Status	Name	Step	Failures	Logs	Raw logs
:heavy_check_mark:	JVM Tests - JDK 11
✖	JVM Tests - JDK 11 Windows	Build	Failures	Logs	Raw logs
:heavy_check_mark:	JVM Tests - JDK 17
:heavy_check_mark:	JVM Tests - JDK 17 MacOS M1
✖	JVM Tests - JDK 18	Build	Failures	Logs	Raw logs

Full information is available in the Build summary check run.

Failures

:gear: JVM Tests - JDK 11 Windows #

- Failing: extensions/opentelemetry/deployment 
! Skipped: integration-tests/micrometer-prometheus integration-tests/opentelemetry integration-tests/opentelemetry-grpc and 5 more

:package: extensions/opentelemetry/deployment

✖ io.quarkus.opentelemetry.deployment.instrumentation.RestClientOpenTelemetryTest.client line 65 - More details - Source on GitHub

org.opentest4j.AssertionFailedError: expected: <SERVER> but was: <CLIENT>
	at org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
	at org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)

---

:gear: JVM Tests - JDK 18 #

- Failing: integration-tests/elytron-undertow 

:package: integration-tests/elytron-undertow

✖ io.quarkus.it.undertow.elytron.BaseAuthRestTest.testPost line 28 - More details - Source on GitHub

java.lang.AssertionError: 
1 expectation failed.
Expected status code <200> but was <500>.

[gsmet]

Hmmm, I think it's too big of a behavioral change to get backported post CR1, especially since I saw some CI failures that might be related to it. Let's give it some bake time.