Error: could not install The Observability Stack: could not install The Observability Stack: rate: Wait(n=1) would exceed context deadline

[Rajakavitha1]

What happened? Unable to install The Observability Stack on a Linode Kubernetes cluster. Did you expect to see something different?

How to reproduce it (as minimally and precisely as possible):

Environment

• tobs installation method:

• [x] cli
• [ ] helm chart

• tobs version:

  tobs version

  Tobs CLI Version: 0.10.0, latest tobs helm chart version: 0.10.1

• Kubernetes version information: 1.23

  kubectl version

kubectl version WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short. Use --output=yaml|json to get the full version. Client Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.0", GitCommit:"4ce5a8954017644c5420bae81d72b09b735c21f0", GitTreeState:"clean", BuildDate:"2022-05-03T13:36:49Z", GoVersion:"go1.18.1", Compiler:"gc", Platform:"darwin/arm64"} Kustomize Version: v4.5.4 Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.6", GitCommit:"ad3338546da947756e8a88aa6822e9c11e7eac22", GitTreeState:"clean", BuildDate:"2022-04-14T08:43:11Z", GoVersion:"go1.17.9", Compiler:"gc", Platform:"linux/amd64"}

• Kubernetes cluster kind:

  insert how you created your cluster: LKE (Linode Kubernetes Engine)

• tobs Logs:

insert tobs logs relevant to the issue here

Anything else we need to know?

cert-manager version v1.6.1 exists in the cluster, skipping the installation...
Installing The Observability Stack, this can take a few minutes
Error: could not install The Observability Stack: could not install The Observability Stack: rate: Wait(n=1) would exceed context deadline
rajakavithakodhandapani@Rajakavithas-MacBook-Pro-2 ~ % tobs version
Tobs CLI Version: 0.10.0, latest tobs helm chart version: 0.10.1 
rajakavithakodhandapani@Rajakavithas-MacBook-Pro-2 ~ % kubectl get pods
NAME                                                        READY   STATUS             RESTARTS         AGE
alertmanager-tobs-kube-prometheus-alertmanager-0            2/2     Running            0                33m
opentelemetry-operator-controller-manager-8cf5c85c8-mw7jz   2/2     Running            0                34m
prometheus-tobs-kube-prometheus-prometheus-0                2/2     Running            0                33m
tobs-connection-secret-7x8wm                                0/1     Completed          0                34m
tobs-grafana-6b6db7ddc9-zgp6g                               2/3     CrashLoopBackOff   11 (53s ago)     34m
tobs-kube-prometheus-operator-f4cbb9f64-mjc25               1/1     Running            0                34m
tobs-kube-state-metrics-765c569cbc-zdx4j                    1/1     Running            0                34m
tobs-prometheus-node-exporter-2hwzc                         1/1     Running            0                34m
tobs-prometheus-node-exporter-kzlcr                         1/1     Running            0                33m
tobs-prometheus-node-exporter-pm9fb                         1/1     Running            0                33m
tobs-promlens-7685c9744d-w5x45                              1/1     Running            0                34m
tobs-promscale-7454f9c99d-hsf4l                             0/1     CrashLoopBackOff   11 (2m10s ago)   34m
tobs-timescaledb-0                                          1/1     Running            0                33m
rajakavithakodhandapani@Rajakavithas-MacBook-Pro-2 ~ % kubectl version
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.0", GitCommit:"4ce5a8954017644c5420bae81d72b09b735c21f0", GitTreeState:"clean", BuildDate:"2022-05-03T13:36:49Z", GoVersion:"go1.18.1", Compiler:"gc", Platform:"darwin/arm64"}
Kustomize Version: v4.5.4
Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.6", GitCommit:"ad3338546da947756e8a88aa6822e9c11e7eac22", GitTreeState:"clean", BuildDate:"2022-04-14T08:43:11Z", GoVersion:"go1.17.9", Compiler:"gc", Platform:"linux/amd64"}```

[ramonguiu]

I deployed on minikube v1.22.0 (K8s v1.21.2 on Docker 20.10.7) on my laptop and also got the same error.

I0520 09:44:50.229344   21225 request.go:665] Waited for 2.133260693s due to client-side throttling, not priority and fairness, request: GET:https://127.0.0.1:55230/api/v1/namespaces/default/services/tobs-promlens
Error: INSTALLATION FAILED: rate: Wait(n=1) would exceed context deadline

The installation succeeds:

kubectl get pods
NAME                                                        READY   STATUS      RESTARTS   AGE
alertmanager-tobs-kube-prometheus-alertmanager-0            2/2     Running     0          3m3s
opentelemetry-operator-controller-manager-7467bdc8b-htczv   2/2     Running     0          5m24s
prometheus-tobs-kube-prometheus-prometheus-0                2/2     Running     0          3m2s
tobs-connection-secret-2lrpr                                0/1     Completed   0          5m24s
tobs-grafana-7db787f487-2zp45                               3/3     Running     2          5m24s
tobs-grafana-db-jfp4m                                       0/1     Completed   0          5m24s
tobs-kube-prometheus-operator-5874b95798-rw69h              1/1     Running     0          5m24s
tobs-kube-state-metrics-56c568fdcc-vrw66                    1/1     Running     0          5m24s
tobs-prometheus-node-exporter-4d986                         1/1     Running     0          5m24s
tobs-promlens-7f778cc958-mp9nk                              1/1     Running     0          5m24s
tobs-promscale-648d58d4f5-g2w5q                             1/1     Running     0          5m24s
tobs-timescaledb-0                                          1/1     Running     0          5m23s

[paulfantom]

The error you are seeing is most likely related to a slow startup of timescaledb pod (due to very large image size). For helm installation method I recommend increasing timeout during installation with helm install --wait --timeout=15m. Sadly there is no such fix available for tobs CLI.

There is an initiative in the company to reduce the image size which would have a direct effect on tobs installation time.

[github-actions[bot]]

This issue went stale because it was not updated in a month. Please consider updating it to improve the quality of the project.

[github-actions[bot]]

This issue was closed because it has been stalled for 30 days with no activity.