tempo icon indicating copy to clipboard operation
tempo copied to clipboard

Why can not see traceID from loki?

Open btdan opened this issue 2 years ago • 3 comments

Hello, I have set up k8s, and inside k8s, The promtail+loki+object sotrage works well. But now I want to see which part cost most time? Then I want to set up the grafana tempo. At first, I see that the config file of loki should be added these environment parameters in the statefulset.yaml : ----------- environment parameters begin ------------------- env: - name: JAEGER_AGENT_HOST value: tempo - name: JAEGER_ENDPOINT value: http://tempo:14268/api/traces - name: JAEGER_SAMPLER_TYPE value: const - name: JAEGER_SAMPLER_PARAM value: "1" ----------- environment parameters end ------------------- After this modification, I run this command to make it in use: kubectl apply -f statefulset.yaml -n loki

And then I query with grafana. However, I can not find any keyword about "traceID" from the query result. Is there anything wrong? Thanks very much.

btdan avatar Apr 22 '22 13:04 btdan

The first thing we should do is simply confirm that Tempo is receiving traces. Two quick ways are to:

  • Check tempo_distributor_spans_received_total and tempo_ingester_traces_created_total to make sure these values are increasing.
  • The distributor has a config option "log_received_traces" for debugging that will simply print every traceid it receives. This is great for confirming that Tempo is receiving traces: https://grafana.com/docs/tempo/latest/configuration/#distributor
distributor:
  log_received_traces: true

joe-elliott avatar Apr 22 '22 14:04 joe-elliott

Thanks very much for your reply. In fact, I want to make clear that if without the tempo, only add the enviroment parameters to the Loki ,there will no data for traceID . Is that right? Thanks very much.

btdan avatar Apr 27 '22 03:04 btdan

Loki is a very powerful way to find trace ids, but Tempo supports search natively as well. These links may help:

https://grafana.com/docs/tempo/latest/operations/backend_search/ https://grafana.com/docs/tempo/latest/getting-started/tempo-in-grafana/#tempo-search

joe-elliott avatar Apr 27 '22 12:04 joe-elliott

This issue has been automatically marked as stale because it has not had any activity in the past 60 days. The next time this stale check runs, the stale label will be removed if there is new activity. The issue will be closed after 15 days if there is no new activity. Please apply keepalive label to exempt this Issue.

github-actions[bot] avatar Nov 15 '22 00:11 github-actions[bot]