quickwit icon indicating copy to clipboard operation
quickwit copied to clipboard

Searching for a traceId in grafana / jaeger does not work after 3 days.

Open uptickmetachu opened this issue 1 year ago • 3 comments

Describe the bug Versions: quickwit v0.8.2 Grafana v11.8.3 jaegertracing/jaeger-query:1.53.0

Datasources:

  • 1 jaeger datasource against jaeger query
  • 1 jaeger datasource against quickwit (api/v1/otel-traces-v0_7/jaeger)

I make a search for a trace in grafana explore against a trace_id and I can see it. A few days later I cannot find the trace via the jaeger search (whereas I can still find it via the quickwit datasource).

I manually test trace Ids and I've isolated that traceIds fail to be searched (via jaeger) exactly after 3 days.

Expected behavior I expect the trace to still be viewable via the quickwit viewer after a few days.

Configuration: Please provide:

  1. Output of quickwit --version
  2. The index_config.yaml

uptickmetachu avatar Dec 09 '24 22:12 uptickmetachu

Actually I found the conversation on discord about jaeger config lookback_period_hours which defaults to 72hours which matches this experience but unfortunately isn't documented anywhere on the website.

uptickmetachu avatar Dec 09 '24 22:12 uptickmetachu

I'm assuming we configure it via:

jager:
   lookback_period_hours: XXX

And inside the helm chart:

config:
  jager:
    lookback_period_hours:

uptickmetachu avatar Dec 09 '24 23:12 uptickmetachu

I guess the intention is to limit the lookback period once a trace has been found. The problem is that this isn't working as intended and is excluding the trace ID if it doens't even land within the lookback period.

uptickmetachu avatar Dec 09 '24 23:12 uptickmetachu