Telegraf unable to scrape localhost:8080/metrics/ endpoint with error 406 Not Acceptable

[danlsgiga]

What went wrong?

What happened: In Telegraf, this line defines a constant value for the Accept header value used by Telegraf's HTTP client. If I do a curl -H "Accept: application/vnd.google.protobuf;proto=io.prometheus.client.MetricFamily;encoding=delimited;q=0.7,text/plain;version=0.0.4;q=0.3" http://localhost:8080/metrics/ I get a 406 Not Acceptable error as well from the Oncall HTTP Server, which is the same error I'm seeing in Telegraf's logs as below:

telegraf[10438]: 2024-06-11T20:14:00Z E! [inputs.prometheus] Error in plugin: "http://localhost:8080/metrics/" returned HTTP status "406 Not Acceptable"

And this is the corresponding log from the Oncall HTTP server:

2024-06-11 22:05:10 source=engine:app google_trace_id=none logger=root inbound latency=0.000662 status=406 method=GET path=/metrics/ user_agent=Telegraf/1.31.0 Go/1.22.4 content-length=0 slow=0
2024-06-11 22:05:10 source=engine:app google_trace_id=none logger=django.request Not Acceptable: /metrics/

If I do a curl http://localhost:8080/metrics/ the request succeeds and I can see the oncall related metrics in the response.

What did you expect to happen: Telegraf to be able to scrape the http://localhost:8080/metrics/ endpoint successfully.

How do we reproduce it?

1. Run the official docker-compose.yml stack with the FEATURE_PROMETHEUS_EXPORTER_ENABLED set to true
2. Curl the http://localhost:8080/metrics/ endpoint using the command in the bug description
3. Verify the HTTP response is 406 Not Acceptable

Grafana OnCall Version

Tested with v1.4.4 and v1.7.0

Product Area

Other

Grafana OnCall Platform?

Docker

User's Browser?

No response

Anything else to add?

No response