Add performance comparision with prometheus-compatible systems

[ernado]

You can use VictoriaMetrics/prometheus-benchmark.

From project README:

Prometheus-benchmark provides the following features:

• It generates production-like workload for both data ingestion and querying paths:
  • It generates write workload from real node_exporter metrics. This is the most frequently used exporter for Prometheus metrics.
  • It generates read workload from typical alerting rules for node_exporter metrics - see chart/files/alerts.yaml.
• It allows generating time series churn rate via scrapeConfigUpdatePercent and scrapeConfigUpdateInterval options. The churn rate is typical for Kubernetes monitoring.
• Multiple systems can be tested simultaneously - just add multiple named entries under remoteStorages section at chart/values.yaml.

The following systems can be tested with prometheus-benchmark:

• Single-node version of VictoriaMetrics:
  • How to push data to VictoriaMetrics
  • How to query data from VictoriaMetrics
• Cluster version of VictoriaMetrics:
  • How to push and query data in cluster version of VictoriaMetrics
• Grafana Mimir:
  • How to push data to Mimir
  • How to query data from Mimir
• Cortex:
  • How to push data to Cortex
  • How to query data from Cortex
• Thanos:
  • How to push data to Thanos

So, list of compatible systems:

• Prometheus
• VictoriaMetrics
• Cortex
• Thanos
• Grafana Mimir

[flyingmutant]

Thanks for the suggestion, an interesting benchmark. Maybe one day we'll find time to run it or similar one, but to be honest I don't see a lot of value.

The things we focus on (multi-level multi-tenancy with fair resource budgeting, low query latency under mixed read-heavy workloads, space-efficient long-term storage with automatic downsampling) are quite different from the things being measured here, and will be much harder to quantatively benchmark. Highlighting only the raw throughput (even though I believe should be excellent in StatsHouse) does not do StatsHouse justice.

[ernado]

Thank you for your repsonse!

I think that it is possible to measure low query latency, space efficiency and down-sampling. System performance under high multi-tenant load should demonstrate statshouse overall superiority and resiliency for overloads.

This will be a pretty strong argument for using statshouse instead of existing systems, people love benchmarks.

I understand that advanced benchmarks are hard to setup, but I think that profit worth the effort.