references (span links) not showing for traces in Grafana

[Sup3rGeo]

Hi,

When we query traces from Clickhouse as a data source we don't see the span references listed, which is different from when I check traces from Tempo, where the reference links are in fact listed in the trace view.

I was able to somewhat fix it by adding another transformation to the sql query:

arrayMap(x,y -> map('spanID', x, 'traceID', y), Links.SpanId, Links.TraceId) AS references

This makes the links appear, but we are missing link attributes which are useful to identify/name each of the links.

Unfortunately trying to add the Links.Attributes column as tags (as it seems to be the case with Tempo) wouldn't work:

arrayMap(x,y,z -> map('spanID', x, 'traceID', y, 'tags', z), Links.SpanId, Links.TraceId, Links.Attributes) AS references

the query fails because clickhouse requires all map values to be of the same type, and in this case we have (String, String, Map).

Is there a way to have Links.Attributes showing up as span references in the Trace view in Grafana? Thanks!

Environment:

• Grafana version: 11.2.0
• Plugin version: 4.3.2

[tatchi]

the query fails because clickhouse requires all map values to be of the same type, and in this case we have (String, String, Map).

I'm hitting the same issue with trace events: https://github.com/grafana/clickhouse-datasource/discussions/840#discussioncomment-10706180. I wonder what we should do in this case ?

[SpencerTorres]

I've experienced this error before with the array types. We have some types that may be able to handle this, but I'm not sure how they behave in an array. Perhaps some form of Variant, Dynamic, or JSON would work. For now you may be able to cast the map to a string by serializing it as JSON via toJSONString. It may not look pretty in the UI, but it wouldn't break.

If we find a nice way to SELECT this we can add it to the default trace UI

[tatchi]

Thanks for your input. It doesn't beak the UI but attributes aren't shown. I believe because it expects an array while it is a string

[tatchi]

I found a solution for events, see https://github.com/grafana/clickhouse-datasource/discussions/840#discussioncomment-10799129. I guess we can apply the same solution for Links.Attributes ?

[SpencerTorres]

Impressive! It can be quite a puzzle sometimes to get the right structure for these Grafana panels. I'll see if we can add it by default

[Sup3rGeo]

Thanks @tatchi ! This makes the span links show properly as references in the UI. While this is not implemented and merged, forked and added it unconditionally in the sqlGenerator.ts file (right before https://github.com/grafana/clickhouse-datasource/blob/main/src/data/sqlGenerator.ts#L152) and built a custom version of the datasource.

I still do have a problem though - when you click the span reference link it does not go to the linked span, it opens the same trace id instead.

I could trace (pun unintended) the problem to the link itself. I decoded the url and the request json I show below. While we have correct trace and span ids at the bottom, "builderOptions.meta.traceId" has the old trace id still. I found out that if I manually replace this with the correct trace id, the link works as expected. The traceid in queries.rawSql does not seem to have any effect on the link navigation.

Maybe someone (@SpencerTorres ?) know whether this is something to be fixed in clickhouse-datasource itself or rather grafana? I know from previous experiments that it works properly with Tempo.

{
  "datasource": "edzok1dhfbb40f",
  "queries": [
    {
      "editorType": "builder",
      "rawSql": "WITH 'b4e6bc9254ba9e14a5e0d16df003e42b' as trace_id, (SELECT min(Start) FROM \"otel\".\"otel_traces_trace_id_ts\" WHERE TraceId = trace_id) as trace_start, (SELECT max(End) + 1 FROM \"otel\".\"otel_traces_trace_id_ts\" WHERE TraceId = trace_id) as trace_end SELECT \"TraceId\" as traceID, \"SpanId\" as spanID, \"ParentSpanId\" as parentSpanID, \"ServiceName\" as serviceName, \"SpanName\" as operationName, multiply(toUnixTimestamp64Nano(\"Timestamp\"), 0.000001) as startTime, multiply(\"Duration\", 0.000001) as duration, arrayMap(key -> map('key', key, 'value',\"SpanAttributes\"[key]), mapKeys(\"SpanAttributes\")) as tags, arrayMap(key -> map('key', key, 'value',\"ResourceAttributes\"[key]), mapKeys(\"ResourceAttributes\")) as serviceTags, arrayMap((spanid, traceid, attributes) -> tuple(spanid, traceid, arrayMap(key -> map('key', key, 'value', attributes[key]), mapKeys(attributes)))::Tuple(spanID String, traceID String, tags Array(Map(String, String))), Links.SpanId, Links.TraceId, Links.Attributes) AS references FROM \"otel\".\"otel_traces\" WHERE traceID = trace_id AND \"Timestamp\" >= trace_start AND \"Timestamp\" <= trace_end LIMIT 1000",
      "builderOptions": {
        "database": "otel",
        "table": "otel_traces",
        "queryType": "traces",
        "mode": "list",
        "columns": [
          {
            "name": "Timestamp",
            "hint": "time"
          },
          {
            "name": "TraceId",
            "hint": "trace_id"
          },
          {
            "name": "SpanId",
            "hint": "trace_span_id"
          },
          {
            "name": "ParentSpanId",
            "hint": "trace_parent_span_id"
          },
          {
            "name": "ServiceName",
            "hint": "trace_service_name"
          },
          {
            "name": "SpanName",
            "hint": "trace_operation_name"
          },
          {
            "name": "Duration",
            "hint": "trace_duration_time"
          },
          {
            "name": "SpanAttributes",
            "hint": "trace_tags"
          },
          {
            "name": "ResourceAttributes",
            "hint": "trace_service_tags"
          }
        ],
        "meta": {
          "minimized": true,
          "isTraceIdMode": true,
          "traceId": "b4e6bc9254ba9e14a5e0d16df003e42b",       <-- THIS POINTS TO THE OLD TRACE ID
          "traceDurationUnit": "nanoseconds",
          "otelEnabled": true,
          "otelVersion": "latest"
        },
        "limit": 1000,
        "filters": [],
        "orderBy": []
      },
      "pluginVersion": "4.4.0",
      "refId": "Trace ID",
      "format": 3,
      "query": "0bd38a1dab067c1f4c99b6bbe6cee443"        <-- CORRECT LINKED TRACE ID
    }
  ],
  "panelsState": {
    "trace": {
      "spanId": "85954949219f076d"                   <-- CORRECT LINKED SPAN ID
    }
  }
}

[ValentinLvr]

Hi ! Getting the same problem. I Managed to add the references in the UI but the redirection is not working (exactly the same error than @Sup3rGeo )

Any updates on this one ?

[ValentinLvr]

Just bumped into this MR that should fix it #1043

[aangelisc]

Hi all, #1208 has been merged which I believe should resolve this. If it doesn't then please reopen this issue. Thanks!

[marcusjsford]

Hi @ValentinLvr, @SpencerTorres

I've come across this same issue using version v4.10.2 of the Clickhouse Datasource. When clicking the "View linked span" it redirects you to the same trace instead of the one from the link.traceid field. The behaviour seen is exactly as described in your comment here on #1208 Which is tagged "waiting-on-grafana". I can't find another issue open that covers this yet.