prometheus Startup Readiness probe keeps on failing. pod Events: Startup probe failed: HTTP probe failed with statuscode: 503

[Mayankm13]

Prometheus Startup Readiness probe keeps on failing. pod Events: Startup probe failed: HTTP probe failed with statuscode: 503

I have increase the timeout but still it's giving

Events: Type Reason Age From Message

---

Normal Scheduled 13m default-scheduler Successfully assigned monitoring/prometheus-k8s-0 to da-cicd-r2-srv-17 Normal SuccessfulAttachVolume 13m attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-f8997b2a-37ab-408a-8c7b-39bd7e150518" Normal Pulled 13m kubelet Container image "quay.io/coreos/configmap-reload:v0.0.1" already present on machine Normal Created 13m kubelet Created container rules-configmap-reloader Normal Pulled 13m kubelet Container image "quay.io/coreos/prometheus-config-reloader:v0.33.0" already present on machine Normal Created 13m kubelet Created container prometheus-config-reloader Normal Started 13m kubelet Started container prometheus-config-reloader Normal Started 13m (x2 over 13m) kubelet Started container prometheus Normal Pulled 13m (x2 over 13m) kubelet Container image "quay.io/prometheus/prometheus:v2.24.1" already present on machine Normal Created 13m (x2 over 13m) kubelet Created container prometheus Normal Started 13m kubelet Started container rules-configmap-reloader Warning Unhealthy 11m (x11 over 13m) kubelet Readiness probe failed: HTTP probe failed with statuscode: 503

apiVersion: apps/v1 kind: StatefulSet metadata: labels: app: prometheus prometheus: k8s name: prometheus-k8s namespace: monitoring ownerReferences:

• apiVersion: monitoring.coreos.com/v1 blockOwnerDeletion: true controller: true kind: Prometheus name: k8s spec: podManagementPolicy: Parallel replicas: 1 revisionHistoryLimit: 10 selector: matchLabels: app: prometheus prometheus: k8s serviceName: prometheus-operated template: metadata: creationTimestamp: null labels: app: prometheus prometheus: k8s spec: containers:
  • args:
    • --web.console.templates=/etc/prometheus/consoles
    • --web.console.libraries=/etc/prometheus/console_libraries
    • --storage.tsdb.retention.size=30GB
    • --config.file=/etc/prometheus/config_out/prometheus.env.yaml
    • --storage.tsdb.path=/prometheus
    • --storage.tsdb.retention.time=15d
    • --web.enable-lifecycle
    • --storage.tsdb.no-lockfile
    • --web.enable-admin-api
    • --web.route-prefix=/ image: quay.io/prometheus/prometheus:v2.24.1 imagePullPolicy: IfNotPresent livenessProbe: failureThreshold: 6 httpGet: path: /-/healthy port: web scheme: HTTP periodSeconds: 5 successThreshold: 1 timeoutSeconds: 3 name: prometheus ports:
    • containerPort: 9090 name: web protocol: TCP readinessProbe: failureThreshold: 120 httpGet: path: /-/ready port: web scheme: HTTP periodSeconds: 10 successThreshold: 1 timeoutSeconds: 15 resources: requests: memory: 2Gi terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts:
    • mountPath: /etc/prometheus/config_out name: config-out readOnly: true
    • mountPath: /prometheus name: prometheus-k8s-db subPath: prometheus-db
    • mountPath: /etc/prometheus/rules/prometheus-k8s-rulefiles-0 name: prometheus-k8s-rulefiles-0
  • args:
    • --log-format=logfmt
    • --reload-url=http://localhost:9090/-/reload
    • --config-file=/etc/prometheus/config/prometheus.yaml.gz
    • --config-envsubst-file=/etc/prometheus/config_out/prometheus.env.yaml command:
    • /bin/prometheus-config-reloader env:
    • name: POD_NAME valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.name image: quay.io/coreos/prometheus-config-reloader:v0.33.0 imagePullPolicy: IfNotPresent name: prometheus-config-reloader resources: limits: cpu: 100m memory: 25Mi terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts:
    • mountPath: /etc/prometheus/config name: config
    • mountPath: /etc/prometheus/config_out name: config-out
  • args:
    • --webhook-url=http://localhost:9090/-/reload
    • --volume-dir=/etc/prometheus/rules/prometheus-k8s-rulefiles-0 image: quay.io/coreos/configmap-reload:v0.0.1 imagePullPolicy: IfNotPresent name: rules-configmap-reloader resources: limits: cpu: 100m memory: 25Mi terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts:
    • mountPath: /etc/prometheus/rules/prometheus-k8s-rulefiles-0 name: prometheus-k8s-rulefiles-0 dnsPolicy: ClusterFirst restartPolicy: Always schedulerName: default-scheduler securityContext: fsGroup: 2000 runAsNonRoot: true runAsUser: 1000 serviceAccount: prometheus serviceAccountName: prometheus terminationGracePeriodSeconds: 600 volumes:
  • name: config secret: defaultMode: 420 secretName: prometheus-k8s
  • emptyDir: {} name: config-out
  • configMap: defaultMode: 420 name: prometheus-k8s-rulefiles-0 name: prometheus-k8s-rulefiles-0 updateStrategy: type: RollingUpdate volumeClaimTemplates:
• apiVersion: v1 kind: PersistentVolumeClaim metadata: annotations: prometheus: k8s creationTimestamp: null name: prometheus-k8s-db spec: accessModes:
  • ReadWriteOnce resources: requests: storage: 50Gi storageClassName: rook-block volumeMode: Filesystem status: phase: Pending

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had any activity in the last 60 days. Thank you for your contributions.

[RapidIE]

I'm running into this same issue on k3s v1.24.2+k3s2 using release-v0.11

Distributor ID: Ubuntu Description: Ubuntu 20.04.4 LTS Release: 20.04 Codename: focal

Interestingly, v0.10 didn't have this problem when I tested it, but I went to v0.11 as it's listed as supported on Kubernetes v1.24.2

Digging into the logs I quickly found what the issue appears to be:

ts=2022-08-22T19:53:02.086Z caller=manager.go:317 level=error component="discovery manager scrape" msg="Cannot create service discovery" err="open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory" type=kubernetes

For some reason the kubernetes bearer token isn't being loaded into the container. I verified by logging into the container and found that the /var/run/secrets/kubernetes.io/serviceaccount/ path doesn't exist. I'm not sure what changed between 0.10 and 0.11 that might cause this behavior.

[Deepak275]

Facing the same issue, with version v2.8.0. Please help ?

[mvandijk1001]

This is probably caused by the service account not auto-mounting the token.

To fix it update your ServiceAccount (change <NAME>):

kubectl patch serviceaccount <NAME> -p '{"automountServiceAccountToken": true}'

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had any activity in the last 60 days. Thank you for your contributions.

[github-actions[bot]]

This issue was closed because it has not had any activity in the last 120 days. Please reopen if you feel this is still valid.