Full table scan in ClickHouse due to HyperDX query missing date filter

[m0nikasingh]

I'm running HyperDX with an external ClickHouse and MongoDB setup. Our ClickHouse instance has the "force_index_by_date" setting enabled, which appears to conflict with some of the queries HyperDX runs.

For example, the following query does not include a time filter, which leads to a full table scan:

SELECT DISTINCT lowCardinalityKeys(arrayJoin(LogAttributes.keys)) AS key 
FROM otel.otel_logs 
LIMIT _CAST(1000, 'Int32') 
FORMAT JSON

Is there a recommended way to address this without disabling force_index_by_date? We'd like to avoid full scans, especially given the volume of data we're working with.

[zdyj3170101136]

it would not search whole table, see rows_max_to_read

[Chihsiao]

Adding a time-range filter would significantly improve performance. Currently, when there are many rows, the CPU remains pinned at 100% for several minutes and eventually times out without returning results.