kubernetes-client
kubernetes-client copied to clipboard
fabric8io
Move to use okhttp4 due to a bug on IPV6 based cluster
The current version of fabric8 Kubernetes client is using okhttp3 library which is affected by bug on hostname verification when connecting to the Kubernetes server API in IPV6 based cluster. The same bug affects even the kubernetes-client Java library: https://github.com/kubernetes-client/java/issues/1289 This issue was fixed in okhttp4 as you can see from this already merged PR: https://github.com/square/okhttp/pull/5889
We are facing this while using fabric8 in the Strimzi operator. There is a workaround as described here https://github.com/strimzi/strimzi-kafka-operator/issues/4002 but it's just a workaround. Is there any plan to update the fabric8 client to the okhttp4 library version?
Isn't okhttp4 based on kotlin? Not sure if it can break any backward compatibility. @manusa @oscerd @iocanel @metacosm WDYT??
So what's the solution you envisage? Back porting the fix about IPV6 from okhttp4 to okhttp3 and then updating fabric8? Right now fabric8 is actually bugged with IPV6 clusters due to this problem in okhttp3 :-(
Bumping to 4.x is something that require effort and time. I'm not sure they will backport the fix on 3.x anyway
Well,, that could be a contribution to backport on okhttp3 if they won't do that. Otherwise, you should list this as a well-known bug on fabric8.
Bumping OkHTTP version to 4.x will not only require a lot of work in the client, but some work on the Kubernetes Client Quarkus extension too (I think).
I agree that we should eventually upgrade to 4.x, new issues are bound to happen in the future. But I'm not sure if the timing is right at the moment, since we need to deliver 5.0 ASAP.
Maybe for the time being contributing a backport of the fix to OkHttp 3.x is more cost-effective WDYT?
Maybe for the time being contributing a backport of the fix to OkHttp 3.x is more cost-effective WDYT?
Are we confident the backport would be accepted, though? Or are we talking about maintaining a 3.x fork until the time when we decide to switch to 4?
My suggestion was just (for now) try to see if the back-port is accepted in the OkHttp project, since it's only a few lines (https://github.com/square/okhttp/pull/5889 - haven't checked branch compatibility though). Maintaining an OkHttp is not what I was implying. That would probably be a much greater effort than upgrading the dependency in the client.
OkHttp 4.x is binary backwards compatible with OkHttp 3.x. The only breaking experience would be if you have kotlin clients building as it's not source compatible. Full description here https://square.github.io/okhttp/upgrading_to_okhttp_4/
It is not a whole new library, clients can upgrade and downgrade (and do, like in Retrofit) since the public API is maintained.
3.14.x is no longer maintained, so you should upgrade to 4.x unless you wanted legacy support with 3.12.12 (Java 7, Android < 5). See https://square.github.io/okhttp/security/
okhttp drags in kotlin runtime,right ? that brings its own set of challenges.
I'm not aware of any actual runtime. It uses a kotlin stdlib library, but nothing like a JVM runtime.
https://mvnrepository.com/artifact/org.jetbrains.kotlin/kotlin-stdlib-jdk8/1.4.21-2
This is around 20kb.
Any progress with this issue? We are facing the same problem in a IPv6 Kubernetes cluster.
This issue has been automatically marked as stale because it has not had any activity since 90 days. It will be closed if no further activity occurs within 7 days. Thank you for your contributions!
![stale[bot] avatar](https://avatars.githubusercontent.com/in/1724?v=4 "Published by a pub.dev verified publisher"){kind=small}
Hi,
is it already set in stone that fabric8 client replaces okhttp 3 with vert.x client? What are the main advantages from moving to vert.x client?
Sometimes I see transitive dependencies of okhttp as a motivation. I´m wondering where that comes from because when comparing the dependencies of io.vertx:vertx-web-client:4.0.3 and com.squareup.okhttp3:okhttp:4.9.1 the dependency trees look like this:
\--- com.squareup.okhttp3:okhttp:4.9.1
+--- com.squareup.okio:okio:2.8.0
| +--- org.jetbrains.kotlin:kotlin-stdlib:1.4.0 -> 1.4.10
| | +--- org.jetbrains.kotlin:kotlin-stdlib-common:1.4.10
| | \--- org.jetbrains:annotations:13.0
| \--- org.jetbrains.kotlin:kotlin-stdlib-common:1.4.0 -> 1.4.10
\--- org.jetbrains.kotlin:kotlin-stdlib:1.4.10 (*)
\--- io.vertx:vertx-web-client:4.0.3
+--- io.vertx:vertx-web-common:4.0.3
| \--- io.vertx:vertx-core:4.0.3
| +--- io.netty:netty-common:4.1.60.Final
| +--- io.netty:netty-buffer:4.1.60.Final
| | \--- io.netty:netty-common:4.1.60.Final
| +--- io.netty:netty-transport:4.1.60.Final
| | +--- io.netty:netty-common:4.1.60.Final
| | +--- io.netty:netty-buffer:4.1.60.Final (*)
| | \--- io.netty:netty-resolver:4.1.60.Final
| | \--- io.netty:netty-common:4.1.60.Final
| +--- io.netty:netty-handler:4.1.60.Final
| | +--- io.netty:netty-common:4.1.60.Final
| | +--- io.netty:netty-resolver:4.1.60.Final (*)
| | +--- io.netty:netty-buffer:4.1.60.Final (*)
| | +--- io.netty:netty-transport:4.1.60.Final (*)
| | \--- io.netty:netty-codec:4.1.60.Final
| | +--- io.netty:netty-common:4.1.60.Final
| | +--- io.netty:netty-buffer:4.1.60.Final (*)
| | \--- io.netty:netty-transport:4.1.60.Final (*)
| +--- io.netty:netty-handler-proxy:4.1.60.Final
| | +--- io.netty:netty-common:4.1.60.Final
| | +--- io.netty:netty-buffer:4.1.60.Final (*)
| | +--- io.netty:netty-transport:4.1.60.Final (*)
| | +--- io.netty:netty-codec:4.1.60.Final (*)
| | +--- io.netty:netty-codec-socks:4.1.60.Final
| | | +--- io.netty:netty-common:4.1.60.Final
| | | +--- io.netty:netty-buffer:4.1.60.Final (*)
| | | +--- io.netty:netty-transport:4.1.60.Final (*)
| | | \--- io.netty:netty-codec:4.1.60.Final (*)
| | \--- io.netty:netty-codec-http:4.1.60.Final
| | +--- io.netty:netty-common:4.1.60.Final
| | +--- io.netty:netty-buffer:4.1.60.Final (*)
| | +--- io.netty:netty-transport:4.1.60.Final (*)
| | +--- io.netty:netty-codec:4.1.60.Final (*)
| | \--- io.netty:netty-handler:4.1.60.Final (*)
| +--- io.netty:netty-codec-http:4.1.60.Final (*)
| +--- io.netty:netty-codec-http2:4.1.60.Final
| | +--- io.netty:netty-common:4.1.60.Final
| | +--- io.netty:netty-buffer:4.1.60.Final (*)
| | +--- io.netty:netty-transport:4.1.60.Final (*)
| | +--- io.netty:netty-codec:4.1.60.Final (*)
| | +--- io.netty:netty-handler:4.1.60.Final (*)
| | \--- io.netty:netty-codec-http:4.1.60.Final (*)
| +--- io.netty:netty-resolver:4.1.60.Final (*)
| +--- io.netty:netty-resolver-dns:4.1.60.Final
| | +--- io.netty:netty-common:4.1.60.Final
| | +--- io.netty:netty-buffer:4.1.60.Final (*)
| | +--- io.netty:netty-resolver:4.1.60.Final (*)
| | +--- io.netty:netty-transport:4.1.60.Final (*)
| | +--- io.netty:netty-codec:4.1.60.Final (*)
| | +--- io.netty:netty-codec-dns:4.1.60.Final
| | | +--- io.netty:netty-common:4.1.60.Final
| | | +--- io.netty:netty-buffer:4.1.60.Final (*)
| | | +--- io.netty:netty-transport:4.1.60.Final (*)
| | | \--- io.netty:netty-codec:4.1.60.Final (*)
| | \--- io.netty:netty-handler:4.1.60.Final (*)
| \--- com.fasterxml.jackson.core:jackson-core:2.11.3
+--- io.vertx:vertx-auth-common:4.0.3
| \--- io.vertx:vertx-core:4.0.3 (*)
\--- io.vertx:vertx-core:4.0.3 (*)
Also from a size point of view, its ~2.6 MB vs ~5.3 MB (results of building a fat jar with just okhttp or vert.x).
I agree that the project should replace okhttp 3 rather sooner than later. And if you prefer vert.x over okhttp, that's perfectly fine. Just interested in the reasons for it :)
One issue is having a dependency that requires a full kotlin build to rebuild from source. (Afaik)
Other is that by using vert.x we can share and optimize http stacks with frameworks like vert.x itself and quarkus.
Optimally the http client layer would be pluggable but not sure how feasible that is.
What do you mean about full Kotlin build? And from source? OkHttp itself is near 100% Kotlin. But it's just a java jar when you use it as a dependency.
@ppatierno: Hello, I'm looking into this issue. I reproduced your issue by using KubernetesClient with a kind cluster with ipv6 configuration specified in kind docs[0]. Are you able to resolve your issue by excluding okhttp3 dependencies introduced by kubernetes-client jar and adding direct okhttp 4 dependencies?
I tested this on a simple reproducer project[1] on a ipv6 kind cluster and it seems to be working fine for me.
<properties>
<fabric8.version>5.5.0</fabric8.version>
<okhttp.version>4.9.0</okhttp.version>
</properties>
<dependencies>
<dependency>
<groupId>io.fabric8</groupId>
<artifactId>kubernetes-client</artifactId>
<version>${fabric8.version}</version>
<exclusions>
<exclusion>
<groupId>com.squareup.okhttp3</groupId>
<artifactId>okhttp</artifactId>
</exclusion>
<exclusion>
<groupId>com.squareup.okhttp3</groupId>
<artifactId>logging-interceptor</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>com.squareup.okhttp3</groupId>
<artifactId>okhttp</artifactId>
<version>${okhttp.version}</version>
</dependency>
<dependency>
<groupId>com.squareup.okhttp3</groupId>
<artifactId>logging-interceptor</artifactId>
<version>${okhttp.version}</version>
</dependency>
</dependencies>
I also checked to upgrade Okhttp in KubernetesClient to v4.9.0:
- https://github.com/rohanKanojia/mockwebserver/commit/60cacde5fce83dedf20d0699cfa58e1464673675
- https://github.com/rohanKanojia/kubernetes-client/commit/96c05159307ae21b13a89549a5bec04714bfa0fa
I was able to get that reproducer project working on Kind ipv6 cluster with 5.6-SNAPSHOT(based on my branch). But I need to test it further to see if it's completely safe to upgrade.
I tested this with Quarkus KubernetesClient Extension as well, I'm able to get it working on a Kubernetes ipv6 cluster with the above exclusions. You can see the full project here.
[0] https://kind.sigs.k8s.io/docs/user/configuration/#ip-family [1] https://github.com/rohankanojia-forks/fabric8-okhttp-ipv6-k8s-cluster-bug-reproducer
Hello community!
I would also be interested in either upgrading okhttp or the migration to Vert.x - whatever is decided upon. Reason: CVE-2021-0341 reported by NexusIQ in okhttp in versions prior to 4.9.1, see https://github.com/square/okhttp/issues/6724 for details.
The upgrade of okhttp to 4.9.1 or the most recent 5.0.0-alpha.2 would be what fixes the CVE but if you decide to move to Vert.x this would also solve the problem.
Who can make the decision or either ways to proceed with eliminating the CVE rather than implementing a workaround (e.g. as was suggested earlier for a different problem https://github.com/fabric8io/kubernetes-client/issues/2632#issuecomment-880089245)?
Thanks
UPD: 4.9.1 version of okhttp does NOT contain yet the necessary fix, only 5.0.0-alpha.2 does.
@ovolynets the CVE isn't exploitable unless you take the HostnameVerifier from OkHttp and use it for other purposes. Unless you are doing something like that which would be really strange, it isn't exploitable at all.
Hi everyone,
I'm very interested if/when it will be possible to get a Kubernetes plugin with proper IPv6 support. Currently most of k8s clusters are built on VPC using IPv6, and I'm not exception here :) I'm trying to implement jenkins with all necessary connectivity with kubernetes, and additionally configuring everything as a code. When I add Kubernetes Credentials Provider plugin (which is based on Kubernetes Client API), I got the following error:
2022-05-13 11:05:42.248+0000 [id=40] INFO c.c.j.p.k.KubernetesCredentialProvider#startWatchingForSecrets: retrieving secrets with selector: jenkins.io/credentials-type, LabelSelector(matchExpressions=[], matchLabels=null, additionalProperties={})
2022-05-13 11:05:42.306+0000 [id=40] SEVERE c.c.j.p.k.KubernetesCredentialProvider#startWatchingForSecrets: Failed to initialise k8s secret provider, secrets from Kubernetes will not be available
javax.net.ssl.SSLPeerUnverifiedException: Hostname fdc0:95c4:2fc::1 not verified:
certificate: sha256/g9NzNCPT5ppq2NamTlMGTfLlXaQLGTUz9PdAhZC9rSQ=
DN: CN=kube-apiserver
subjectAltNames: [fdc0:95c4:2fc:0:0:0:0:1, XXXX:XXXX:XXXX:XXXX:0:0:0:YYYY, 10.0.56.117, fd131eabd68e9bb05d6353da5a27a257.gr7.eu-central-1.eks.amazonaws.com, ip-10-0-56-117.eu-central-1.compute.internal, kubernetes, kubernetes.default, kubernetes.default.svc, kubernetes.default.svc.cluster.local]
at okhttp3.internal.connection.RealConnection.connectTls(RealConnection.java:334)
at okhttp3.internal.connection.RealConnection.establishProtocol(RealConnection.java:284)
at okhttp3.internal.connection.RealConnection.connect(RealConnection.java:169)
at okhttp3.internal.connection.StreamAllocation.findConnection(StreamAllocation.java:258)
at okhttp3.internal.connection.StreamAllocation.findHealthyConnection(StreamAllocation.java:135)
at okhttp3.internal.connection.StreamAllocation.newStream(StreamAllocation.java:114)
at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:42)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:127)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
at io.fabric8.kubernetes.client.okhttp.OkHttpClientBuilderImpl$InteceptorAdapter.intercept(OkHttpClientBuilderImpl.java:62)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
at io.fabric8.kubernetes.client.okhttp.OkHttpClientBuilderImpl$InteceptorAdapter.intercept(OkHttpClientBuilderImpl.java:62)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
at io.fabric8.kubernetes.client.okhttp.OkHttpClientBuilderImpl$InteceptorAdapter.intercept(OkHttpClientBuilderImpl.java:62)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
at io.fabric8.kubernetes.client.okhttp.OkHttpClientBuilderImpl$InteceptorAdapter.intercept(OkHttpClientBuilderImpl.java:62)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:257)
at okhttp3.RealCall.execute(RealCall.java:93)
at io.fabric8.kubernetes.client.okhttp.OkHttpClientImpl.send(OkHttpClientImpl.java:138)
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.retryWithExponentialBackoff(OperationSupport.java:574)
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:553)
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:518)
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:502)
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.listRequestHelper(BaseOperation.java:133)
Caused: io.fabric8.kubernetes.client.KubernetesClientException: Operation: [list] for kind: [Secret] with name: [null] in namespace: [azalejski] failed.
at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:130)
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.listRequestHelper(BaseOperation.java:140)
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.list(BaseOperation.java:415)
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.list(BaseOperation.java:404)
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.list(BaseOperation.java:83)
at com.cloudbees.jenkins.plugins.kubernetes_credentials_provider.KubernetesCredentialProvider.startWatchingForSecrets(KubernetesCredentialProvider.java:116)
at com.cloudbees.jenkins.plugins.kubernetes_credentials_provider.KubernetesCredentialProvider$1.doRun(KubernetesCredentialProvider.java:161)
at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:92)
at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:67)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
2022-05-13 11:05:42.307+0000 [id=40] INFO c.c.j.p.k.KubernetesCredentialProvider#reconnectLater: Attempting to reconnect Kubernetes client in 5 mins
The most important thing here is IPv6 address [fdc0:95c4:2fc::1], which probably can't be successfully verified.
I found temporary solution for this topic setting environment variable KUBERNETES_DISABLE_HOSTNAME_VERIFICATION=true in jenkins container, but in my opinion the whole host verification mechanism should be implemented with IPv6 working properly.
Best regards, Arek
Removed the never stale - it doesn't seem like an okhttp4 based option is in the cards to be distributed with the core project. It's easy enough for a user to create one on their own, and especially with 6.3 the jetty and jdk clients are viable alternatives. Vertx should come along in whatever follows 6.3.
Well, one of the options I was considering is to (finally) upgrade to OkHttp 4 once the Vert.x client is out and Quarkus is no longer conditioned by it. In any case, that should be tackled before this issue becomes stale again :)
This issue has been automatically marked as stale because it has not had any activity since 90 days. It will be closed if no further activity occurs within 7 days. Thank you for your contributions!
Are there still planes to upgrade to okhttp4? the 3.x version depends on an older junit dependency from kubernetes-server-mock through mockwebserver of okhttp3. And this gives CVE vulnerability warning in the IDE.
@manusa Kubernetes client does not rely on okhttp anymore - it's uses vertx.
Mockserver using okhttp is unfortunate. Not sure what the cure is here. For now you can manually override the version.
Hi @maxandersen
@manusa Kubernetes client does not rely on okhttp anymore - it's uses vertx.
I'm not sure if your intention was to tag me here or you wanted to reply to @tomdw instead.
Fabric8 Kubernetes Client relies on OkHttp for its kubernetes-httpclient-okhttp HttpClient implementation and the Kubernetes Mock Web Server.
@tomdw, as Max said, you should be able to override the OkHttp version for MockWebServer by forcing the dependency to a newer OkHttp version like documented here.
Mockserver using okhttp is unfortunate. Not sure what the cure is here.
Our kubernetes-server-mock and mockwebserver modules rely on OkHttp's mockwebserver as its backend.
One of the options I was considering was to replace this with a Vertx or a Jetty server. However, this will require additional work since right now everything is tightly coupled to OkHttp. (discussed in https://github.com/fabric8io/kubernetes-client/issues/5613)