Error: "INVALID" is not a valid start token

[ghost]

Env description

BlackBox and prometheus are runs on kubernetes Blackbox container : prom/blackbox-exporter:v0.17.0 Prometheus container : prom/prometheus:v2.20.1 Blackbox runs in replicaSet of 5 and has k8s service on front of it.

After running a few days some of url health checks are failing and in prometheus logs we see following message

level=debug ts=2020-09-10T23:56:26.331Z caller=scrape.go:1009 component="scrape manager" scrape_pool=blackbox target="http://blackbox:80/probe?module=http_2xx&target=https%3A%2F%2Fapi-sandbox.mydomain.net%2F" msg="Append failed" err="\"INVALID\" is not a valid start token"
level=debug ts=2020-09-10T23:56:26.339Z caller=scrape.go:1009 component="scrape manager" scrape_pool=blackbox target="http://blackbox:80/probe?module=http_2xx&target=https%3A%2F%2Fapi-sandbox.mydomain.net%2F" msg="Append failed" err="\"INVALID\" is not a valid start token"

on prometheus interface we will see "INVALID" is not a valid start token

After blackbox restart all starts working as expected

Resource utilization are below 20% of request memory and cpu

What is the blackbox.yml module config.

modules:
  http_2xx:
    prober: http
  http_post_2xx:
    prober: http
    http:
      method: POST
  tcp_connect:
    prober: tcp
  pop3s_banner:
    prober: tcp
    tcp:
      query_response:
      - expect: "^+OK"
      tls: true
      tls_config:
        insecure_skip_verify: false
  ssh_banner:
    prober: tcp
    tcp:
      query_response:
      - expect: "^SSH-2.0-"
  irc_banner:
    prober: tcp
    tcp:
      query_response:
      - send: "NICK prober"
      - send: "USER prober prober prober :prober"
      - expect: "PING :([^ ]+)"
        send: "PONG ${1}"
      - expect: "^:[^ ]+ 001"
  icmp:
    prober: icmp

What is the prometheus.yml scrape config.

global:
  scrape_interval:     15s # By default, scrape targets every 15 seconds.
remote_write:
- url: http://mycortex.devops.local/api/prom/push
scrape_configs:
  - job_name: 'prometheus'
    scrape_interval: 15s
    scheme: http
    static_configs:
      - targets: ['blackbox:80']
  - job_name: 'blackbox'
    scheme: http
    scrape_interval: 15s
    metrics_path: /probe
    params:
      module: [http_2xx]
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: blackbox:80
    static_configs:
      - targets:
        - https://front.mydomain.com
        - https://api.front.mydomain.com/handshake?ns=check
        labels:
          team: digitalMarketing
          env: prod
      - targets:
        - https://api-dev.mydomain.net
        - https://api-sandbox.mydomain.net
        labels:
          team: nginx
          env: prod

What did you do that produced an error?

After running a few days error comes up

What did you expect to see?

Expect to run without blackbox restarts

[brian-brazil]

That shouldn't be possible, can you get a tcpdump of what the blackbox exporter is sending back?

[ghost]

@brian-brazil Thanks for response I will add --log.level=debug to blackbox exporter (As tcpdump will be a bit challenging on k8s cluster with istio) To see if this will give any details. And it's also takes time to reproduce. Will update you as I get more details

[djamaile]

Did you find a work around or solution?

[ghost]

No we didn't. Potential problem can be that we are using istio however we never faced any problem

[electron0zero]

author of this issue has deleted the account, so I will close this.

If anyone is facing this issue in lastest version, please create a new issue :)