Slow loki label drop down loading on Grafana

[qqu0127]

HI,

Background: I deployed qryn with a clickhouse proxy. The ingestion load is about 2MB/s. In my setup, qryn is added as loki source on Grafana and that's where we mostly use for log query.

Problem: Using the Grafana UI, I notice it takes very long time for the label dropdown options to load. Specifically, it works fine for the first label search, it seems both the label name and values are pre-fetched or indexed. However, when it comes to the second and more label filtering it takes long time just to load the label options. It timeout and cause OOM in the qryn app. The second or more label filter is basically not working for me. I know I can use loki native query directly but that's simply not a good choice.

My findings so far: I found the qryn log:

WITH sel_a AS (select samples.string as string,samples.fingerprint as fingerprint,samples.timestamp_ns as timestamp_ns from qryn_cluster.samples_v3_dist as samples where ((samples.timestamp_ns between 1724798773369000000 and 1724809573369000000) and (samples.type in (0,0))) and (samples.fingerprint IN (select sel_1.fingerprint from (select fingerprint from qryn_cluster.time_series_gin where ((key = 'namespace') and (val = 'my_namespace'))) as sel_1)) order by timestamp_ns desc limit 10) select JSONExtractKeysAndValues(time_series.labels, 'String') as labels,sel_a.* from sel_a left any join qryn_cluster.time_series_dist AS time_series on sel_a.fingerprint = time_series.fingerprint order by labels desc,timestamp_ns desc

Not a SQL expert, but I feel this one is inefficient. Anyone has a clue?