History query failed: error="failed to run a range query Prometheus for alerts: bad_data: invalid parameter \"start\"

[bimax]

Hello,

I am running karma in a docker: docker run -p 8080:8080 -v $(pwd)/karma-config/karma.yaml:/etc/karma/karma.yaml ghcr.io/prymitive/karma:latest --config.file=/etc/karma/karma.yaml

Versions: karma: latest Alertmanager: 0.27.0 Prometheus: 2.50.1 in dc1 and 2.42.0 in dc2

with a config file:

alertmanager:
  interval: 2m
  servers:
    - name: ams.monitoring-system
      uri: https://alertmanager.k8s-dc1.domain.net
      readonly: false
      proxy: false
    - name: ams.prod
      uri: https://alertmanager-prod.k8s-dc1.domain.net
      readonly: false
      proxy: false
    - name: sjc.monitoring-system
      uri: https://alertmanager.k8s-dc2.domain.net
      readonly: false
      proxy: false
    - name: sjc.prod
      uri: https://alertmanager-prod.k8s-dc2.domain.net
      readonly: false
      proxy: false
alertAcknowledgement:
  enabled: true
  duration: 15m0s
  author: karma
  comment: ACK! This alert was acknowledged using karma on %NOW%

And I see batch of errors in the logs:

level=error msg="History query failed" error="failed to run a range query Prometheus for alerts: bad_data: invalid parameter \"start\": cannot parse \"\" to a valid timestamp" labels={"alertname":"KubeQuotaFullyUsed","namespace":"son-qa"} uri=http://prometheus.k8s-dc2.domain.net worker=2
level=error msg="History query failed" error="failed to run a range query Prometheus for alerts: bad_data: invalid parameter \"start\": cannot parse \"\" to a valid timestamp" labels={"alertname":"KubeQuotaFullyUsed","container":"kube-state-metrics","endpoint":"http","namespace":"haproxy-ingress","prometheus":"monitoring-system/kps-kube-prometheus-stack-prometheus","resource":"services.loadbalancers","retention":"2y","service":"kps-kube-state-metrics","severity":"info","target":"slack-internal"} uri=http://prometheus.k8s-dc1.domain.net worker=30
...

Not sure what is wrong with it. Can you please advice.

[jhuspek]

Hi, I had a same issue. The problem was that karma wanted to access prometheus via http and catches redirect to https. So the solution was to add a rewrite rule to replace http with https.