tempo icon indicating copy to clipboard operation
tempo copied to clipboard

Tempo (v2.5.0) Distributor missing tempo_receiver_accepted_spans

Open jameshounshell opened this issue 8 months ago • 3 comments

Describe the bug Today I upgraded the tempo-distributed helm chart from 1.8.5 to 1.10.0 (app version 2.3.1 to 2.5.0) and I noticed in the Tempo Operational dashboard, that the tempo_receiver_accepted_spans metric disappeared. I'm assuming this the result to a change in the embedded OTLP library but other than the change noted in the release, there was nothing specific about this metric, making me think breaking it was unintentional.

To Reproduce Steps to reproduce the behavior:

  1. Install version 1.10.0 of the tempo-distributed helm chart.
  2. Verify the absence of tempo_receiver_accepted_spans at /metrics of the tempo distributor component.

Expected behavior I would expect the metric tempo_receiver_accepted_spans to still exist/be-emitted.

Environment:

  • Infrastructure: GKE fully managed
  • Deployment tool: Kustomize plus helm (helmCharts) chart

Additional Context Here is a graph showing tempo_receiver_ metrics disappearing while tempo_distributor_spans_received_total remains. image

jameshounshell avatar Jun 07 '24 14:06 jameshounshell