promscale icon indicating copy to clipboard operation
promscale copied to clipboard

Support collection of raw measurements

Open oliora opened this issue 3 years ago • 0 comments

Promscale provides a useful abstraction on top of TimescaleDB in area of managing labels, series and exemplars and implementing heavily optimized bulk inserts because of caching, temporary tables and other optimization tricks. It would be great if one can store raw measurements with Promscale in the way similar to metrics. There is no widely used protocol for this that I'm aware of so designing one may be a part of the story.

There is my scenario for raw measurements:

  1. Application emits raw measurement of latency that contains a timestamp, metric name, latency value in nanoseconds, set of attributes that define a dimension (series) and set of attributes that define an exemplar and stores it in the file.
  2. Data collector reads the file, batches raw measurements and writes them to TimescaleDB in the set of tables which structure is similar to Promscale metric tables with the difference that there is no unique index on the measurements data table because there may be several measurements collected within the same time interval for the same series.
  3. In the database there is a materialized view for each measurement that aggregates measurements into a histogram. The view structure is the same as the metrics view of Promscale.

I see that for my scenario I can reuse a lot of things from Promscale (to repeat myself: managing of labels, series and exemplars and optimized inserts) and I'd like to do so.

oliora avatar Jul 28 '22 09:07 oliora