promscale_extension icon indicating copy to clipboard operation
promscale_extension copied to clipboard

Optimize queries in the drop_metric_chunks path

Open sumerman opened this issue 2 years ago • 0 comments

While working on https://github.com/timescale/promscale/pull/1634, we discovered that some queries on the metrics retention code path could be significantly sped up.

WITH conf AS MATERIALIZED (SELECT _prom_catalog.get_default_retention_period() AS def_retention)
SELECT count(*)
FROM _timescaledb_catalog.dimension_slice ds
         INNER JOIN _timescaledb_catalog.dimension d ON (d.id = ds.dimension_id)
         INNER JOIN _timescaledb_catalog.hypertable h ON (h.id = d.hypertable_id)
         INNER JOIN _prom_catalog.metric m ON (m.table_name = h.table_name AND m.table_schema = h.schema_name)
         JOIN conf ON TRUE
WHERE ds.range_start < _timescaledb_internal.time_to_internal(now() - coalesce(m.retention_period, conf.def_retention))
  AND ds.range_end < _timescaledb_internal.time_to_internal(now() - coalesce(m.retention_period, conf.def_retention));

Unlike the naïve query, based on show_chunks, this seems to fall consistently under 70ms on a test system regardless of the number of chunks it discovers.

The naïve query for comparison, which is, roughly, what the maintenance job does presently:

SELECT count(sc.sc)
FROM _prom_catalog.metric m,
     LATERAL _prom_catalog.get_default_retention_period() ret_p,
     LATERAL _prom_catalog.get_storage_hypertable_info(m.table_schema, m.table_name, m.is_view) hi,
     LATERAL public.show_chunks(hi.hypertable_relation,
                                older_than=>coalesce(m.retention_period,
                                                     ret_p)) sc;

It takes from 30ms to over a second, depending on the number of chunks it counts.

sumerman avatar Sep 12 '22 09:09 sumerman