Grafana Agent still send exemplar metric after I set `send_exemplars` to false

[christopherzli]

Grafana Agent still send exemplar metric after I set send_exemplars to false; slack thread for context: https://grafana.slack.com/archives/C01050C3D8F/p1696582638649219

We used grafana agent to write to our M3DB Coordinator. However, we recieved

url=http://localhost:9900/api/v1/prom/remote/write msg="non-recoverable error" count=1999 exemplarCount=1 err="server returned HTTP status 400 Bad Request: {\"status\":\"error\",\"error\":\"proto: wrong wireType = 2 for field Type\"}"

because of conflicts of proto def. Specifically, https://github.com/prometheus/prometheus/blob/main/prompb/types.proto#L128 grafana agent is trying to send exemplars in protobuf, however the definition is different for 3 in Timeseries in M3DB: https://github.com/databricks/m3/blob/db_main/src/query/generated/proto/prompb/types.proto#L17 I started a thread with M3DB to fix their proto. However, after I set send_exemplars to false in remote_write config, I still see the same error message of exemplar metrics is trying to our M3DB backend. We are currently using v0.35 and observed this issue. We just upgraded from v0.15 where send_exemplars is default to false. We would like to know if it could be possibly due to send_exemplars not correctly parsed in and applied in metric ingestion path. Could you point me to where you check send_exemplars? I did not find any reference in grafana agent repo.

[hainenber]

Can you give out the agent's config that encountering this issue? Thanks!

[rfratto]

Hi there :wave:

On April 9, 2024, Grafana Labs announced Grafana Alloy, the spirital successor to Grafana Agent and the final form of Grafana Agent flow mode. As a result, Grafana Agent has been deprecated and will only be receiving bug and security fixes until its end-of-life around November 1, 2025.

To make things easier for maintainers, we're in the process of migrating all issues tagged variant/flow to the Grafana Alloy repository to have a single home for tracking issues. This issue is likely something we'll want to address in both Grafana Alloy and Grafana Agent, so just because it's being moved doesn't mean we won't address the issue in Grafana Agent :)

[mattdurham]

@christopherzli can you check and see if this is still occurring? I was unable to recreate this.