jkube:kubernetes using depricated extensions/v1beta1 for deployment

[burads]

Description

I have been converting a maven project using JKube to use Kubernetes with Rancher rather then OpenShift. So have been switching components to use jkube:kubernetes-maven-plugin rather then jkube:openshift-maven-plugin

When doing k8s:resource it is now generating the correct ingress, service, deployment and configmap that i can all apply using kubectl.

But when trying to use k8s:apply jkube reverts to using extensions/v1beta1 for the deployment rather then apps/v1 which is defined in deplyment.yaml.

This results in an error:

[ERROR] Failed to execute goal org.eclipse.jkube:kubernetes-maven-plugin:1.7.0:apply on project demo-app: 
Execution of goal org.eclipse.jkube:kubernetes-maven-plugin:1.7.0:apply failed: 
Failure executing: GET at: https://clusterurl.example.com/k8s/clusters/c-1234/apis/apps/v1/namespaces/test-sandbox/deployments/demo-app-dev. 
Message: deployments.extensions "demo-app-dev" is forbidden: 
User "u-sample" cannot get resource "deployments" in API group "extensions" in the namespace "test-sandbox". 
Received status: Status(apiVersion=v1, code=403, details=StatusDetails(causes=[], group=extensions, kind=deployments, name=demo-app-dev, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=deployments.extensions "demo-app-dev" is forbidden: User "u-sample" cannot get resource "deployments" in API group "extensions" in the namespace "test-sandbox", metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=Forbidden, status=Failure, additionalProperties={}). 
-> [Help 1]

An additional note here is that jkube is actually incorrect about the get url. When looking at the access log of my api endpoint it states jkube was trying to get from "https://clusterurl.example.com/k8s/clusters/c-1234/apis/extensions/v1beta1/namespaces/test-sandbox/deployments/demo-app-dev" not the url in the Error message.

Info

• Eclipse JKube version : 1.7.0

• Maven version (mvn -v) : 3.8.4

• Kubernetes / Red Hat OpenShift setup and version : v1.21.2

• If it's a bug, how to reproduce : mvn clean k8s:resource k8s:apply

[DEBUG] Goal: org.eclipse.jkube:kubernetes-maven-plugin:1.7.0:apply [DEBUG] Style: Regular [DEBUG] Configuration:

<configuration>
  <access>
    <masterUrl>clusterurl.example.com</masterUrl>
    <namespace>test-sandbox</namespace>
  </access>
  <createNewResources default-value="true">${jkube.deploy.create}</createNewResources>
  <deletePodsOnReplicationControllerUpdate default-value="true">${jkube.deploy.deletePods}</deletePodsOnReplicationControllerUpdate>
  <environment>dev</environment>
  <failOnNoKubernetesJson default-value="false">${jkube.deploy.failOnNoKubernetesJson}</failOnNoKubernetesJson>
  <ignoreRunningOAuthClients default-value="true">${jkube.deploy.ignoreRunningOAuthClients}</ignoreRunningOAuthClients>
  <ignoreServices default-value="false">${jkube.deploy.ignoreServices}</ignoreServices>
  <jsonLogDir default-value="${basedir}/target/jkube/applyJson">${jkube.deploy.jsonLogDir}</jsonLogDir>
  <kubernetesManifest default-value="${basedir}/target/classes/META-INF/jkube/kubernetes.yml">${jkube.kubernetesManifest}</kubernetesManifest>
  <mojoExecution default-value="${mojoExecution}"/>
  <namespace>test-sandbox</namespace>
  <offline default-value="false">${jkube.offline}</offline>
  <processTemplatesLocally default-value="false">${jkube.deploy.processTemplatesLocally}</processTemplatesLocally>
  <project default-value="${project}"/>
  <recreate default-value="false">${jkube.recreate}</recreate>
  <resourceDir default-value="${basedir}/src/main/jkube">${jkube.resourceDir}</resourceDir>
  <resources>
    <labels>
      <service>
        <name>demo-app</name>
      </service>
    </labels>
    <replicas>1</replicas>
    <annotations>
      <ingress>
        <property>
          <name>cert-manager.io/cluster-issuer</name>
          <value>vault-issuer</value>
        </property>
        <property>
          <name>cert-manager.io/common-name</name>
          <value>demo-app.example.com</value>
        </property>
      </ingress>
    </annotations>
  </resources>
  <rollingUpgrades default-value="false">${jkube.rolling}</rollingUpgrades>
  <serviceUrlWaitTimeSeconds default-value="5">${jkube.serviceUrl.waitSeconds}</serviceUrlWaitTimeSeconds>
  <servicesOnly default-value="false">${jkube.deploy.servicesOnly}</servicesOnly>
  <session default-value="${session}"/>
  <settings default-value="${settings}"/>
  <skip default-value="false">${jkube.skip}</skip>
  <skipApply default-value="false">${jkube.skip.apply}</skipApply>
  <useColor default-value="true">${jkube.useColor}</useColor>
  <verbose default-value="false">${jkube.verbose}</verbose>
</configuration>

[manusa]

Thanks for your report, we'll take a look into this. It seems to be definitely a bug.

[ghost]

Have been trying to continue debugging and trying to reduce complelxity

Trying to deploy into a vanilla K3s Cluster with jkube works fine. Adding rancher on top and using rancher cluster API also works fine.

When I switch the project to use our AKS cluster with the rancher API it results in this error.

I considered if it was maybe AKS / Rancher that did some mangeling with the get requst. So tried basic curl. which results in the expected 404 on the correkt endpoint. Not the forbidden extentions error from above.

curl https://clusterurl.example.com/k8s/clusters/c-1234/apis/apps/v1/namespaces/test-sandbox/deployments/demo-app-dev -H "Authorization: Bearer @satoken"
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {

  },
  "status": "Failure",
  "message": "deployments.apps \"demo-app-dev\" not found",
  "reason": "NotFound",
  "details": {
    "name": "demo-app-dev",
    "group": "apps",
    "kind": "deployments"
  },
  "code": 404
}

I can only reproduce the issue if it curl the same api endpoint from my log:

curl https://clusterurl.example.com/k8s/clusters/c-1234/apis/extensions/v1beta1/namespaces/test-sandbox/deployments/demo-app -H "Authorization: Bearer @satoken"
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {

  },
  "status": "Failure",
  "message": "deployments.extensions \"demo-app-dev\" is forbidden: User \"u-sample\" cannot get resource \"deployments\" in API group \"extensions\" in the namespace \"test-sandbox\"",
  "reason": "Forbidden",
  "details": {
    "name": "demo-app-dev",
    "group": "extensions",
    "kind": "deployments"
  },
  "code": 403
}

So it seems like there is some communication earlier with the cluser that makes jkube select "extensions/v1beta1" rather then "apps/v1". But this error only occur when it is AKS and not k3s.

But I don't get any log output with other get requests even with mvn -X -e

[ghost]

Even more digging.

This seem to be related to an cluster policy. After tracing communication with kubectl proxy -v7 i found some very interesting traces and a difference between my K3S cluster and the AKS Cluster.

As part of the process of deploying it tries to find the state of the cluster.

PUT https://clusterurl.example.com/k8s/clusters/c-1234/api/v1/namespaces/test-sandbox/configmaps/demo-app-config - Response Status: 200 OK
GET https://clusterurl.example.com/k8s/clusters/c-1234/apis/apps/v1/namespaces/test-sandbox/deployments/demo-app-dev - Response Status: 404 Not Found
GET https://clusterurl.example.com/k8s/clusters/c-1234/apis/extensions/v1beta1/namespaces/test-sandbox/deployments/demo-app-dev -Response Status: 403 Forbidden

PUT https://k3sclusterurl.example.com/k8s/clusters/c-1234/api/v1/namespaces/test-sandbox/configmaps/demo-app-config - Response Status: 200 OK
GET https://k3sclusterurl.example.com/k8s/clusters/c-1234/apis/apps/v1/namespaces/test-sandbox/deployments/demo-app-dev - Response Status: 404 Not Found
GET https://k3sclusterurl.example.com/k8s/clusters/c-1234/apis/extensions/v1beta1/namespaces/test-sandbox/deployments/demo-app-dev -Response Status: 404 Not Found

So maybe it is an RBAC policy or similar that we have on our AKS cluster that results in the 403. But would expect that jkube should just continue after the 403 rather then be blocked by it when it is only the "fallback" from a test if it exists.

[manusa]

Might be related to https://github.com/fabric8io/kubernetes-client/issues/4226 and the BackwardsCompatibilityInterceptor kicking in.

Could you try setting the system property kubernetes.backwardsCompatibilityInterceptor.disable or environment variable KUBERNETES_BACKWARDSCOMPATIBILITYINTERCEPTOR_DISABLE to true?

[rohanKanojia]

@burads : polite ping, Did you get any chance to revisit this? Does it work for you when you set kubernetes.backwardsCompatibilityInterceptor.disable ?

[rohanKanojia]

I'm not sure what action to take here. We have upgraded to Fabric8 Kubernetes Client v6.0.0 in https://github.com/eclipse/jkube/pull/1464 . BackwardsCompatibilityInterceptor should be disabled by default now.

[manusa]

I'm not sure what action to take here. We have upgraded to Fabric8 Kubernetes Client v6.0.0 in #1464 . BackwardsCompatibilityInterceptor should be disabled by default now.

With the client upgrade, the behavior changes, so there's no action from our side here. Just wait for feedback after we release 1.9.0.

("fixed" by #1464)

[ghost]

I "unfortunately" found a workaround the issue was RBAC policies that are very locked down for normal users. This also means it is only whitelisted components that are allowed access to.

So I have had our clusteradmin allow access to api_groups = ["extensions"] resources = ["deployments"] verbs = ["create", "delete", "get", "list", "watch"] which made jkube able to both apply and delete.

This changed the 403 to 404 for our internal cluster.

So I unfortunately have a bit of a hard time testing it. But should be reproduceable by testing with a user where extensions are disallowed.

[manusa]

Interceptor should now be disabled by default in v1.9.0 which we just released.

[manusa]

I'm closing this issue as fixed, please reopen if the error persists.