series and label table entries are deleted when data expires

[franck102]

After time series data is deleted according to retention policies, the _prom_catalog.delete_expired_series deletes what appears to be unused series & labels entries.

A common use case however is to compute time-based aggregates on the raw data (e.g. daily / monthly totals), and to keep those aggregates for much longer than the raw data (which is not needed anymore once the aggregates are available). Those aggregates still refer to the series & label tables however, so it would be nice to have some configuration control over expiration policies for those.

[Harkishen-Singh]

Thank you for reporting this. I don't think this is a priority but will discuss it. Also, making deletion of expired series optional is not good either, since multiple promscale instances can have conflicts. Not sure about the aggs thing here.

cc @cevian

[franck102]

I don't understand the conflict issue with multiple promscale instances - series_id are not reused as far as I can tell?

Maintaining aggregates in the long-term when their metadata is expired is really hard - that would probably mean introducing triggers to make a copy of labels & of their values as they appear :(

[VineethReddy02]

Hey @franck102

We are working on a technical spike for automatic metric roll-ups in Promscale. I see you are already using aggregates on your end. Do you mind providing more details on your use cases and how you are aggregating data today?