kubernetes-client
kubernetes-client copied to clipboard
fabric8io
Java generator from CRD: class not found in quarkus when enabling extraAnnotations
Describe the bug
When generating Java classes from CRD within Quarkus I get ClassNotFound Exceptions.
I was playing around with annotation processing settings but nothing helped.
Every time I enable the extraAnnotations I get exceptions when executing. It is also taking way longer to generate and I even generates the class io/fabric8/kubernetes/api/model/ObjectMetaBuilder that was not found in the target folder.
Fabric8 Kubernetes Client version
7.0.0, 6.13.14
Steps to reproduce
POM:
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>test</groupId>
<artifactId>test-component</artifactId>
<version>1.0.0-SNAPSHOT</version>
<properties>
<compiler-plugin.version>3.13.0</compiler-plugin.version>
<maven.compiler.release>21</maven.compiler.release>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
<quarkus.platform.artifact-id>quarkus-bom</quarkus.platform.artifact-id>
<quarkus.platform.group-id>io.quarkus.platform</quarkus.platform.group-id>
<quarkus.platform.version>3.17.3</quarkus.platform.version>
<kubernetes-client.version>7.0.0</kubernetes-client.version>
<sundrio.version>0.200.0</sundrio.version>
<lombok.version>1.18.36</lombok.version>
<skipITs>true</skipITs>
</properties>
<dependencyManagement>
<dependencies>
<dependency>
<groupId>${quarkus.platform.group-id}</groupId>
<artifactId>${quarkus.platform.artifact-id}</artifactId>
<version>${quarkus.platform.version}</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>
<dependencies>
<dependency>
<groupId>io.quarkus</groupId>
<artifactId>quarkus-kubernetes-client</artifactId>
</dependency>
<dependency>
<groupId>io.quarkus</groupId>
<artifactId>quarkus-arc</artifactId>
</dependency>
<!-- extraAnnotations requires these additional dependencies -->
<dependency>
<groupId>io.sundr</groupId>
<artifactId>builder-annotations</artifactId>
<version>${sundrio.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
<version>${lombok.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>io.fabric8</groupId>
<artifactId>generator-annotations</artifactId>
</dependency>
<dependency>
<groupId>io.quarkus</groupId>
<artifactId>quarkus-junit5</artifactId>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>${quarkus.platform.group-id}</groupId>
<artifactId>quarkus-maven-plugin</artifactId>
<version>${quarkus.platform.version}</version>
<extensions>true</extensions>
<executions>
<execution>
<goals>
<goal>build</goal>
<goal>generate-code</goal>
<goal>generate-code-tests</goal>
<goal>native-image-agent</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<artifactId>maven-compiler-plugin</artifactId>
<version>${compiler-plugin.version}</version>
<configuration>
<parameters>true</parameters>
<annotationProcessorPaths>
<path>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
<version>${lombok.version}</version>
</path>
<path>
<groupId>io.sundr</groupId>
<artifactId>builder-annotations</artifactId>
<version>${sundrio.version}</version>
</path>
<path>
<groupId>io.fabric8</groupId>
<artifactId>generator-annotations</artifactId>
<version>${kubernetes-client.version}</version>
</path>
</annotationProcessorPaths>
</configuration>
</plugin>
<plugin>
<groupId>io.fabric8</groupId>
<artifactId>java-generator-maven-plugin</artifactId>
<version>${kubernetes-client.version}</version>
<configuration>
<source>${project.basedir}/test-crd.yaml</source>
<alwaysPreserveUnknown>true</alwaysPreserveUnknown>
<extraAnnotations>true</extraAnnotations>
<generatedAnnotations>true</generatedAnnotations>
</configuration>
<executions>
<execution>
<goals>
<goal>generate</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>
test-crd.yaml: I took one of the PrinterColmn from the tests. But it doesn't matter.
src/test/java/MyTest.java:
import org.junit.jupiter.api.Test;
import io.fabric8.kubernetes.client.KubernetesClient;
import io.quarkus.test.junit.QuarkusTest;
import jakarta.inject.Inject;
@QuarkusTest
public class MyTest {
@Inject
private KubernetesClient kubernetesClient;
@Test
public void test() {
System.out.println("pod list:");
var pods = this.kubernetesClient.pods().list();
pods.getItems().forEach(pod -> {
System.out.println(pod.getMetadata().getName());
});
}
}
mvn clean test
This causes
13:34:40,619 [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 18.49 s <<< FAILURE! -- in MyTest
13:34:40,620 [ERROR] MyTest.test -- Time elapsed: 1.204 s <<< ERROR!
io.fabric8.kubernetes.client.KubernetesClientException: io/fabric8/kubernetes/api/model/ObjectMetaBuilder
at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.waitForResult(OperationSupport.java:509)
at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.list(BaseOperation.java:451)
at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.list(BaseOperation.java:419)
at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.list(BaseOperation.java:98)
at MyTest.test(MyTest.java:15)
at java.base/java.lang.reflect.Method.invoke(Method.java:580)
at io.quarkus.test.junit.QuarkusTestExtension.runExtensionMethod(QuarkusTestExtension.java:967)
at io.quarkus.test.junit.QuarkusTestExtension.interceptTestMethod(QuarkusTestExtension.java:817)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
Caused by: java.lang.NoClassDefFoundError: io/fabric8/kubernetes/api/model/ObjectMetaBuilder
at java.base/java.lang.Class.getDeclaredMethods0(Native Method)
at java.base/java.lang.Class.privateGetDeclaredMethods(Class.java:3578)
at java.base/java.lang.Class.getDeclaredMethods(Class.java:2676)
at com.fasterxml.jackson.databind.util.ClassUtil.getClassMethods(ClassUtil.java:1317)
at com.fasterxml.jackson.databind.introspect.AnnotatedMethodCollector._addMemberMethods(AnnotatedMethodCollector.java:115)
at com.fasterxml.jackson.databind.introspect.AnnotatedMethodCollector.collect(AnnotatedMethodCollector.java:54)
And it seems the class was even generated
Expected behavior
I would think that I can generate the code and run the kubernetes client afterwards.
Runtime
Kubernetes (vanilla)
Kubernetes API Server version
1.25.3@latest
Environment
Linux
Fabric8 Kubernetes Client Logs
No response
Additional context
No response
Hi @spike83 thanks for your interest!
My gut feeling is that there is an incompatibility with the sundrio version, but need to verify it.
Would you be able to share a test-crd.yaml? (possibly a minimal one)
Or a reproducer project that we can play with?
Oh, wait, you are trying to use kubernetes-client at version 7.0.0 with Quarkus that doesn't support it just yet ref.
I doubt this configuration will work cc. @manusa
I tried to use 6.13.4 and get the same error.
Caused by: java.lang.ClassNotFoundException: io.fabric8.kubernetes.api.model.ObjectMetaBuilder
Ok, thanks for the additional info, would you mind providing the CRD content? Or a full reproducer 🙏
@spike83 thanks, unfortunately I cannot reproduce the error using the zip you provided.
The error message I obtain is the expectd one:
io.fabric8.kubernetes.client.KubernetesClientException: The timeout of 10000 ms has been exceeded when getting a connection to my.cluster:6443
at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.waitForResult(OperationSupport.java:509)
at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.list(BaseOperation.java:451)
at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.list(BaseOperation.java:419)
at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.list(BaseOperation.java:98)
at MyTest.test(MyTest.java:15)
at java.base/java.lang.reflect.Method.invoke(Method.java:580)
at io.quarkus.test.junit.QuarkusTestExtension.runExtensionMethod(QuarkusTestExtension.java:967)
at io.quarkus.test.junit.QuarkusTestExtension.interceptTestMethod(QuarkusTestExtension.java:817)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
Caused by: io.vertx.core.impl.NoStackTraceTimeoutException: The timeout of 10000 ms has been exceeded when getting a connection to my.cluster:6443
We need to bisect further the differences:
- java version - I tested with 23 - 21
- Maven version - I'm using 3.9.9
Or have a reproducible reproducer in, for example, a GH Action.
The only thing I'm noticing is that the sundrio version is not aligned with the one used by the k8s client. It should be: https://github.com/fabric8io/kubernetes-client/blob/6ee190c3a36ea6ae5ed7513942e2cabef177a0eb/pom.xml#L129C21-L129C23
instead of 0.200.0, but I can't reproduce the issue with either versions.
I understand that there is no Kubernetes cluster at your address. Could you point to one, as you can see in the test it only tries to list the pods? I guess it's happening when deserializing the actual response. I tried to put a dummy address and get no error.
Here I also have another reproducer which tries to create a pod first and then tries to list the pods. This can be run against the local dev-servcie with
KUBECONFIG=./none.yaml mvn test. This causes another exception of the same nature.
This is what I get here with the second reproducer:
024-12-18 12:28:17,574 INFO [tc.reg.k8s.io/.31.0] (build-9) Creating container for image: registry.k8s.io/kube-apiserver:v1.31.0
2024-12-18 12:28:17,585 INFO [tc.tes.11.0] (build-9) Creating container for image: testcontainers/ryuk:0.11.0
2024-12-18 12:28:17,670 INFO [tc.tes.11.0] (build-9) Container testcontainers/ryuk:0.11.0 is starting: b29291f98b598f34853a664992a9101a7495e26083e3559f944ee0ef9d97d96a
2024-12-18 12:28:17,991 INFO [tc.tes.11.0] (build-9) Container testcontainers/ryuk:0.11.0 started in PT0.405975703S
2024-12-18 12:28:18,106 INFO [tc.reg.k8s.io/.31.0] (build-9) Container registry.k8s.io/kube-apiserver:v1.31.0 is starting: c69b2a8d1924461042224533ce6f4fa505c6e31c5e9737a0e39fb3518b38cf4f
2024-12-18 12:28:18,501 INFO [tc.reg.k8s.io/.5.12-0] (build-9) Creating container for image: registry.k8s.io/etcd:3.5.12-0
2024-12-18 12:28:18,548 INFO [tc.reg.k8s.io/.5.12-0] (build-9) Container registry.k8s.io/etcd:3.5.12-0 is starting: 0177f8f296fb8f2eb137b595cec73426bebc7ec2c45f517dc99065ed47a94fd5
2024-12-18 12:28:18,702 INFO [tc.reg.k8s.io/.5.12-0] (build-9) Container registry.k8s.io/etcd:3.5.12-0 started in PT0.200892865S
2024-12-18 12:28:18,702 INFO [com.daj.kin.ApiServerContainer] (build-9) Waiting for API server...
2024-12-18 12:28:20,588 INFO [tc.reg.k8s.io/.31.0] (build-9) Container registry.k8s.io/kube-apiserver:v1.31.0 started in PT3.014359281S
Dez. 18, 2024 12:28:20 PM io.quarkus.kubernetes.client.deployment.DevServicesKubernetesProcessor setupKubernetesDevService
INFO: Dev Services for Kubernetes started. Other Quarkus applications in dev mode will find the cluster automatically.
Dez. 18, 2024 12:28:21 PM io.quarkus.bootstrap.runner.Timing printStartupTime
INFO: test-component 1.0.0-SNAPSHOT on JVM (powered by Quarkus 3.17.3) started in 10.776s. Listening on: http://localhost:8081
Dez. 18, 2024 12:28:21 PM io.quarkus.bootstrap.runner.Timing printStartupTime
INFO: Profile test activated.
Dez. 18, 2024 12:28:21 PM io.quarkus.bootstrap.runner.Timing printStartupTime
INFO: Installed features: [cdi, kubernetes-client, rest, smallrye-context-propagation, vertx]
12:28:22,187 [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 14.86 s <<< FAILURE! -- in MyTest
12:28:22,188 [ERROR] MyTest.test -- Time elapsed: 0.033 s <<< ERROR!
java.lang.NoClassDefFoundError: io/fabric8/kubernetes/api/model/ObjectMetaFluent
at java.base/java.lang.ClassLoader.defineClass1(Native Method)
at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1027)
at io.quarkus.bootstrap.classloading.QuarkusClassLoader.loadClass(QuarkusClassLoader.java:558)
at io.quarkus.bootstrap.classloading.QuarkusClassLoader.loadClass(QuarkusClassLoader.java:517)
at io.fabric8.kubernetes.api.model.PodFluent.withNewMetadata(PodFluent.java:88)
at MyTest.test(MyTest.java:20)
at java.base/java.lang.reflect.Method.invoke(Method.java:580)
at io.quarkus.test.junit.QuarkusTestExtension.runExtensionMethod(QuarkusTestExtension.java:967)
at io.quarkus.test.junit.QuarkusTestExtension.interceptTestMethod(QuarkusTestExtension.java:817)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
Caused by: java.lang.ClassNotFoundException: io.fabric8.kubernetes.api.model.ObjectMetaFluent
at io.quarkus.bootstrap.classloading.QuarkusClassLoader.loadClass(QuarkusClassLoader.java:542)
at io.quarkus.bootstrap.classloading.QuarkusClassLoader.loadClass(QuarkusClassLoader.java:517)
... 11 more
Dez. 18, 2024 12:28:22 PM io.quarkus.bootstrap.runner.Timing printStopTime
INFO: test-component stopped in 0.040s
Dez. 18, 2024 12:28:22 PM io.quarkus.devservices.common.ContainerShutdownCloseable close
INFO: Dev Services for kubernetes-client shut down.
12:28:22,584 [INFO]
12:28:22,585 [INFO] Results:
12:28:22,585 [INFO]
12:28:22,585 [ERROR] Errors:
12:28:22,585 [ERROR] MyTest.test:20 » NoClassDefFound io/fabric8/kubernetes/api/model/ObjectMetaFluent
I got the reproducer to work and minimized a little further here.
@manusa would you mind taking a look? My memory regarding this flow starts to be rusty ...
I can reproduce the issue with the provided example.
@andreaTP, have we tried the Java Generator with Quarkus before? do you know if there's an integration test somewhere?
Also, maybe the problem might ring a bell to @metacosm
The problem seems to be with the Quarkus test runner and the loading of the generated classes. I'd assume that this won't happen with a regular JUnit test.
have we tried the Java Generator with Quarkus before?
I used the Java generator with the QOSDK a few times.
do you know if there's an integration test somewhere?
I don't think we have specific tests, the Java Generator is emitting POJOs, most of the hiccups we found are related to lombok or sundrio interactions when "extraAnnotations" is enabled.
The problem seems to be with the Quarkus test runner and the loading of the generated classes. I'd assume that this won't happen with a regular JUnit test.
Should we report the bug in the Quarkus repo? We can likely minimize it further to avoid using the java generator at all.
Should we report the bug in the Quarkus repo? We can likely minimize it further to avoid using the java generator at all.
This does seem like a Quarkus bug. We'll probably need to take care of this ourselves, but from what I can see the problem needs fixing in Quarkus.
This issue has been automatically marked as stale because it has not had any activity since 90 days. It will be closed if no further activity occurs within 7 days. Thank you for your contributions!
![stale[bot] avatar](https://avatars.githubusercontent.com/in/1724?v=4 "Published by a pub.dev verified publisher"){kind=small}
I updated to latest the dependencies in the reproducer and is still valid.
Any update on this issue? Is there an issue on the quarkus repo that we can follow? Thanks for your work!
Hi @vanny96 , I just tested and it reproduces with Quarkus 3.23.0.
I believe that this issue belongs to this effort: https://github.com/orgs/quarkusio/projects/30/views/1
Maybe better if we open an issue on the Quarkus side and close this one?