kube-prometheus

kube-prometheus copied to clipboard

What happened?

Delete the namespace after deploying kube-state-metrics gets stuck with the message:

Discovery failed for some groups, 1 failing: unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request

Did you expect to see some different?

Delete should succeed.

How to reproduce it (as minimally and precisely as possible):

jsonnetfile.json:

{
    "dependencies": [
        {
            "name": "kube-prometheus",
            "source": {
                "git": {
                    "remote": "https://github.com/coreos/kube-prometheus",
                    "subdir": "jsonnet/kube-prometheus"
                }
            },
            "version": "release-0.2"
        }
    ]
}

monitoring.jsonnet:

local kp =
  (import 'kube-prometheus/kube-prometheus.libsonnet') +
  {
    _config+:: {
      namespace: 'monitoring',
    },
  };

{ ['00namespace-' + name]: kp.kubePrometheus[name] for name in std.objectFields(kp.kubePrometheus) } +
{ ['kube-state-metrics-' + name]: kp.kubeStateMetrics[name] for name in std.objectFields(kp.kubeStateMetrics) }

Make sure to first disable the metrics-server:

$ minikube addons disable metrics-server

Then the usual convert to gojsontoyaml and kubectl apply...

Once deployed try and delete the namespace:

kubectl delete namespace monitoring

It will get stuck and never finish deleting. Terminating the process and then running:

$ kubectl get namespace monitoring -o json
{
    "apiVersion": "v1",
    "kind": "Namespace",
    "metadata": {
        "annotations": {
            "kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"v1\",\"kind\":\"Namespace\",\"metadata\":{\"annotations\":{},\"name\":\"monitoring\"}}\n"
        },
        "creationTimestamp": "2019-10-22T10:40:11Z",
        "name": "monitoring",
        "resourceVersion": "6480",
        "selfLink": "/api/v1/namespaces/monitoring",
        "uid": "ef064fe3-6903-4362-9f34-d2f11742d89c"
    },
    "spec": {
        "finalizers": [
            "kubernetes"
        ]
    },
    "status": {
        "conditions": [
            {
                "lastTransitionTime": "2019-10-22T10:44:25Z",
                "message": "Discovery failed for some groups, 1 failing: unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request",
                "reason": "DiscoveryFailed",
                "status": "True",
                "type": "NamespaceDeletionDiscoveryFailure"
            },
            {
                "lastTransitionTime": "2019-10-22T10:44:25Z",
                "message": "All legacy kube types successfully parsed",
                "reason": "ParsedGroupVersions",
                "status": "False",
                "type": "NamespaceDeletionGroupVersionParsingFailure"
            },
            {
                "lastTransitionTime": "2019-10-22T10:44:25Z",
                "message": "All content successfully deleted",
                "reason": "ContentDeleted",
                "status": "False",
                "type": "NamespaceDeletionContentFailure"
            }
        ],
        "phase": "Terminating"
    }
}

Environment

• Prometheus Operator version:

N/A

• Kubernetes version information:

Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.0", GitCommit:"2bd9643cee5b3b3a5ecbd3af49d09018f0773c77", GitTreeState:"clean", BuildDate:"2019-09-19T13:57:45Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.0", GitCommit:"2bd9643cee5b3b3a5ecbd3af49d09018f0773c77", GitTreeState:"clean", BuildDate:"2019-09-18T14:27:17Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}

• Kubernetes cluster kind:

minikube

• Manifests:

From Jsonnet library.

• Prometheus Operator Logs:

N/A

• Prometheus Logs:

N/A

Anything else we need to know?:

This is because of a somewhat cyclic dependency of the APIService pointing at the in your situation already deleted prometheus-adapter. If you delete the APIService called metrics.k8s.io/v1beta1 then this should be able to complete.

https://stackoverflow.com/questions/62442679/could-not-get-apiversions-from-kubernetes-unable-to-retrieve-the-complete-list

This could be solved by installing APIServices into a different namespace. I like kube-system for this because that's not going to be deleted until the cluster is going away - but some people might not have permission for that so making it configurable is nice.

#740 addresses this when installing prometheus adapter.

kubectl delete APIServices v1beta1.metrics.k8s.io

tks, ti work well!