Prometheus adapter error "apiserver was unable to write a JSON response: http2: stream closed"

[alexandre-allard]

Component: monitoring, kubernetes

What happened: There are some errors in both prometheus-adapter and in kube-apiserver pod logs.

In apiserver logs:

E0118 13:42:30.917466       1 controller.go:114] loading OpenAPI spec for "v1beta1.custom.metrics.k8s.io" failed with: OpenAPI spec does not exist

In prometheus-adapter logs:

I0118 13:42:29.917414       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (10.334736ms) 200 [Go-http-client/2.0 10.233.68.64:43925]
E0118 13:42:29.917614       1 writers.go:163] apiserver was unable to write a JSON response: http2: stream closed
E0118 13:42:29.917649       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http2: stream closed"}

What was expected: No errors in logs.

Steps to reproduce: Deploy MetalK8s 2.7 or later (maybe it happens on previous versions also, I didn't check).

Resolution proposal (optional): Apparently this is known bug on K8s side (fixed in 1.20), I found the following issues: https://github.com/DirectXMan12/k8s-prometheus-adapter/issues/292 https://github.com/kubernetes/kubernetes/issues/90434

We can disable the default rules in prometheus-adapter chart:

diff --git a/charts/prometheus-adapter.yaml b/charts/prometheus-adapter.yaml
index 53c9a1151..6c2c2936a 100644
--- a/charts/prometheus-adapter.yaml
+++ b/charts/prometheus-adapter.yaml
@@ -10,6 +10,7 @@ prometheus:
 replicas: 1

 rules:
+  default: false
   resource:
     cpu:
       containerQuery: sum(rate(container_cpu_usage_seconds_total{<<.LabelMatchers>>}[3m])) by (<<.GroupBy>>)

So the error disappear, but it does not really fix the issue as it only get rid of the faulty feature. Plus, without this feature we can't use kubectl top anymore.

[alexandre-allard]

It looks like due to this issue, prometheus-adapter is consuming a lot of resources (and the logging stack also)

[thomasdanan]

Should be closed as soon as MetalK8s 2.9 is released