Solr 8.11 with SolrMetrics produces duplicate samples with prometheus v2.52

Open perosb opened this issue 1 year ago • 0 comments
Solr 8.11 seem to be producing duplicate metrics which is flooding the logs and triggering alerts.
This is only when using latest prometheus v2.52.0. Related https://github.com/prometheus/prometheus/issues/14089 This doesn't seem to happen with Solr 9. It still happens when running solr 8.11 and solr-exporter image tag 9.x
This is the debug log with 1 collection and the related duplicates:
{"caller":"scrape.go:1777","component":"scrape manager","level":"debug","msg":"Duplicate sample for timestamp","scrape_pool":"serviceMonitor/solr-test/solr-metrics/0","series":"solr_metrics_core_query_requests_total{category=\"QUERY\",searchHandler=\"/select\",internal=\"false\",core=\"kermit_shard1_replica_n1\",collection=\"kermit\",shard=\"shard1\",replica=\"replica_n1\",base_url=\"http://frog-solrcloud-1.frog-solrcloud-headless.solr-test:8983/solr\",cluster_id=\"f7122eee5d\"}","target":"http://172.23.50.53:8080/metrics","ts":"2024-05-15T08:32:26.909Z"}
{"caller":"scrape.go:1777","component":"scrape manager","level":"debug","msg":"Duplicate sample for timestamp","scrape_pool":"serviceMonitor/solr-test/solr-metrics/0","series":"solr_metrics_core_query_mean_rate{category=\"QUERY\",searchHandler=\"/select\",internal=\"false\",core=\"kermit_shard1_replica_n1\",collection=\"kermit\",shard=\"shard1\",replica=\"replica_n1\",base_url=\"http://frog-solrcloud-1.frog-solrcloud-headless.solr-test:8983/solr\",cluster_id=\"f7122eee5d\"}","target":"http://172.23.50.53:8080/metrics","ts":"2024-05-15T08:32:26.909Z"}
{"caller":"scrape.go:1777","component":"scrape manager","level":"debug","msg":"Duplicate sample for timestamp","scrape_pool":"serviceMonitor/solr-test/solr-metrics/0","series":"solr_metrics_core_query_mean_rate{category=\"QUERY\",searchHandler=\"/select\",internal=\"false\",core=\"kermit_shard1_replica_n1\",collection=\"kermit\",shard=\"shard1\",replica=\"replica_n1\",base_url=\"http://frog-solrcloud-1.frog-solrcloud-headless.solr-test:8983/solr\",cluster_id=\"f7122eee5d\"}","target":"http://172.23.50.53:8080/metrics","ts":"2024-05-15T08:32:26.909Z"}
{"caller":"scrape.go:1777","component":"scrape manager","level":"debug","msg":"Duplicate sample for timestamp","scrape_pool":"serviceMonitor/solr-test/solr-metrics/0","series":"solr_metrics_core_query_5minRate{category=\"QUERY\",searchHandler=\"/select\",internal=\"false\",core=\"kermit_shard1_replica_n1\",collection=\"kermit\",shard=\"shard1\",replica=\"replica_n1\",base_url=\"http://frog-solrcloud-1.frog-solrcloud-headless.solr-test:8983/solr\",cluster_id=\"f7122eee5d\"}","target":"http://172.23.50.53:8080/metrics","ts":"2024-05-15T08:32:26.909Z"}
{"caller":"scrape.go:1777","component":"scrape manager","level":"debug","msg":"Duplicate sample for timestamp","scrape_pool":"serviceMonitor/solr-test/solr-metrics/0","series":"solr_metrics_core_query_1minRate{category=\"QUERY\",searchHandler=\"/select\",internal=\"false\",core=\"kermit_shard1_replica_n1\",collection=\"kermit\",shard=\"shard1\",replica=\"replica_n1\",base_url=\"http://frog-solrcloud-1.frog-solrcloud-headless.solr-test:8983/solr\",cluster_id=\"f7122eee5d\"}","target":"http://172.23.50.53:8080/metrics","ts":"2024-05-15T08:32:26.909Z"}
{"caller":"scrape.go:1777","component":"scrape manager","level":"debug","msg":"Duplicate sample for timestamp","scrape_pool":"serviceMonitor/solr-test/solr-metrics/0","series":"solr_metrics_core_query_p75_ms{category=\"QUERY\",searchHandler=\"/select\",internal=\"false\",core=\"kermit_shard1_replica_n1\",collection=\"kermit\",shard=\"shard1\",replica=\"replica_n1\",base_url=\"http://frog-solrcloud-1.frog-solrcloud-headless.solr-test:8983/solr\",cluster_id=\"f7122eee5d\"}","target":"http://172.23.50.53:8080/metrics","ts":"2024-05-15T08:32:26.909Z"}
{"caller":"scrape.go:1777","component":"scrape manager","level":"debug","msg":"Duplicate sample for timestamp","scrape_pool":"serviceMonitor/solr-test/solr-metrics/0","series":"solr_metrics_core_query_p75_ms{category=\"QUERY\",searchHandler=\"/select\",internal=\"false\",core=\"kermit_shard1_replica_n1\",collection=\"kermit\",shard=\"shard1\",replica=\"replica_n1\",base_url=\"http://frog-solrcloud-1.frog-solrcloud-headless.solr-test:8983/solr\",cluster_id=\"f7122eee5d\"}","target":"http://172.23.50.53:8080/metrics","ts":"2024-05-15T08:32:26.909Z"}
{"caller":"scrape.go:1777","component":"scrape manager","level":"debug","msg":"Duplicate sample for timestamp","scrape_pool":"serviceMonitor/solr-test/solr-metrics/0","series":"solr_metrics_core_query_median_ms{category=\"QUERY\",searchHandler=\"/select\",internal=\"false\",core=\"kermit_shard1_replica_n1\",collection=\"kermit\",shard=\"shard1\",replica=\"replica_n1\",base_url=\"http://frog-solrcloud-1.frog-solrcloud-headless.solr-test:8983/solr\",cluster_id=\"f7122eee5d\"}","target":"http://172.23.50.53:8080/metrics","ts":"2024-05-15T08:32:26.909Z"}
{"caller":"scrape.go:1777","component":"scrape manager","level":"debug","msg":"Duplicate sample for timestamp","scrape_pool":"serviceMonitor/solr-test/solr-metrics/0","series":"solr_metrics_core_query_median_ms{category=\"QUERY\",searchHandler=\"/select\",internal=\"false\",core=\"kermit_shard1_replica_n1\",collection=\"kermit\",shard=\"shard1\",replica=\"replica_n1\",base_url=\"http://frog-solrcloud-1.frog-solrcloud-headless.solr-test:8983/solr\",cluster_id=\"f7122eee5d\"}","target":"http://172.23.50.53:8080/metrics","ts":"2024-05-15T08:32:26.909Z"}
{"caller":"scrape.go:1777","component":"scrape manager","level":"debug","msg":"Duplicate sample for timestamp","scrape_pool":"serviceMonitor/solr-test/solr-metrics/0","series":"solr_metrics_core_requests_total{category=\"QUERY\",handler=\"/select\",internal=\"false\",core=\"kermit_shard1_replica_n1\",collection=\"kermit\",shard=\"shard1\",replica=\"replica_n1\",base_url=\"http://frog-solrcloud-1.frog-solrcloud-headless.solr-test:8983/solr\",cluster_id=\"f7122eee5d\"}","target":"http://172.23.50.53:8080/metrics","ts":"2024-05-15T08:32:26.909Z"}
{"caller":"scrape.go:1777","component":"scrape manager","level":"debug","msg":"Duplicate sample for timestamp","scrape_pool":"serviceMonitor/solr-test/solr-metrics/0","series":"solr_metrics_core_requests_total{category=\"ADMIN\",handler=\"/admin/ping\",internal=\"false\",core=\"kermit_shard1_replica_n1\",collection=\"kermit\",shard=\"shard1\",replica=\"replica_n1\",base_url=\"http://frog-solrcloud-1.frog-solrcloud-headless.solr-test:8983/solr\",cluster_id=\"f7122eee5d\"}","target":"http://172.23.50.53:8080/metrics","ts":"2024-05-15T08:32:26.909Z"}
{"caller":"scrape.go:1777","component":"scrape manager","level":"debug","msg":"Duplicate sample for timestamp","scrape_pool":"serviceMonitor/solr-test/solr-metrics/0","series":"solr_metrics_core_query_p99_ms{category=\"QUERY\",searchHandler=\"/select\",internal=\"false\",core=\"kermit_shard1_replica_n1\",collection=\"kermit\",shard=\"shard1\",replica=\"replica_n1\",base_url=\"http://frog-solrcloud-1.frog-solrcloud-headless.solr-test:8983/solr\",cluster_id=\"f7122eee5d\"}","target":"http://172.23.50.53:8080/metrics","ts":"2024-05-15T08:32:26.910Z"}
{"caller":"scrape.go:1777","component":"scrape manager","level":"debug","msg":"Duplicate sample for timestamp","scrape_pool":"serviceMonitor/solr-test/solr-metrics/0","series":"solr_metrics_core_query_p99_ms{category=\"QUERY\",searchHandler=\"/select\",internal=\"false\",core=\"kermit_shard1_replica_n1\",collection=\"kermit\",shard=\"shard1\",replica=\"replica_n1\",base_url=\"http://frog-solrcloud-1.frog-solrcloud-headless.solr-test:8983/solr\",cluster_id=\"f7122eee5d\"}","target":"http://172.23.50.53:8080/metrics","ts":"2024-05-15T08:32:26.910Z"}
{"caller":"scrape.go:1777","component":"scrape manager","level":"debug","msg":"Duplicate sample for timestamp","scrape_pool":"serviceMonitor/solr-test/solr-metrics/0","series":"solr_metrics_core_query_p95_ms{category=\"QUERY\",searchHandler=\"/select\",internal=\"false\",core=\"kermit_shard1_replica_n1\",collection=\"kermit\",shard=\"shard1\",replica=\"replica_n1\",base_url=\"http://frog-solrcloud-1.frog-solrcloud-headless.solr-test:8983/solr\",cluster_id=\"f7122eee5d\"}","target":"http://172.23.50.53:8080/metrics","ts":"2024-05-15T08:32:26.910Z"}
{"caller":"scrape.go:1777","component":"scrape manager","level":"debug","msg":"Duplicate sample for timestamp","scrape_pool":"serviceMonitor/solr-test/solr-metrics/0","series":"solr_metrics_core_query_p95_ms{category=\"QUERY\",searchHandler=\"/select\",internal=\"false\",core=\"kermit_shard1_replica_n1\",collection=\"kermit\",shard=\"shard1\",replica=\"replica_n1\",base_url=\"http://frog-solrcloud-1.frog-solrcloud-headless.solr-test:8983/solr\",cluster_id=\"f7122eee5d\"}","target":"http://172.23.50.53:8080/metrics","ts":"2024-05-15T08:32:26.910Z"}
{"caller":"scrape.go:1777","component":"scrape manager","level":"debug","msg":"Duplicate sample for timestamp","scrape_pool":"serviceMonitor/solr-test/solr-metrics/0","series":"solr_metrics_node_requests_total{category=\"ADMIN\",handler=\"/admin/info\",base_url=\"http://frog-solrcloud-1.frog-solrcloud-headless.solr-test:8983/solr\",cluster_id=\"f7122eee5d\"}","target":"http://172.23.50.53:8080/metrics","ts":"2024-05-15T08:32:26.910Z"}
{"caller":"scrape.go:1777","component":"scrape manager","level":"debug","msg":"Duplicate sample for timestamp","scrape_pool":"serviceMonitor/solr-test/solr-metrics/0","series":"solr_metrics_node_requests_total{category=\"ADMIN\",handler=\"/admin/metrics\",base_url=\"http://frog-solrcloud-1.frog-solrcloud-headless.solr-test:8983/solr\",cluster_id=\"f7122eee5d\"}","target":"http://172.23.50.53:8080/metrics","ts":"2024-05-15T08:32:26.910Z"}
{"caller":"scrape.go:1777","component":"scrape manager","level":"debug","msg":"Duplicate sample for timestamp","scrape_pool":"serviceMonitor/solr-test/solr-metrics/0","series":"solr_metrics_node_time_seconds_total{category=\"ADMIN\",handler=\"/admin/info\",base_url=\"http://frog-solrcloud-1.frog-solrcloud-headless.solr-test:8983/solr\",cluster_id=\"f7122eee5d\"}","target":"http://172.23.50.53:8080/metrics","ts":"2024-05-15T08:32:26.910Z"}
{"caller":"scrape.go:1777","component":"scrape manager","level":"debug","msg":"Duplicate sample for timestamp","scrape_pool":"serviceMonitor/solr-test/solr-metrics/0","series":"solr_metrics_node_time_seconds_total{category=\"ADMIN\",handler=\"/admin/metrics\",base_url=\"http://frog-solrcloud-1.frog-solrcloud-headless.solr-test:8983/solr\",cluster_id=\"f7122eee5d\"}","target":"http://172.23.50.53:8080/metrics","ts":"2024-05-15T08:32:26.910Z"}
{"caller":"scrape.go:1777","component":"scrape manager","level":"debug","msg":"Duplicate sample for timestamp","scrape_pool":"serviceMonitor/solr-test/solr-metrics/0","series":"solr_metrics_core_time_seconds_total{category=\"QUERY\",handler=\"/select\",internal=\"false\",core=\"kermit_shard1_replica_n1\",collection=\"kermit\",shard=\"shard1\",replica=\"replica_n1\",base_url=\"http://frog-solrcloud-1.frog-solrcloud-headless.solr-test:8983/solr\",cluster_id=\"f7122eee5d\"}","target":"http://172.23.50.53:8080/metrics","ts":"2024-05-15T08:32:26.910Z"}
{"caller":"scrape.go:1777","component":"scrape manager","level":"debug","msg":"Duplicate sample for timestamp","scrape_pool":"serviceMonitor/solr-test/solr-metrics/0","series":"solr_metrics_core_time_seconds_total{category=\"ADMIN\",handler=\"/admin/ping\",internal=\"false\",core=\"kermit_shard1_replica_n1\",collection=\"kermit\",shard=\"shard1\",replica=\"replica_n1\",base_url=\"http://frog-solrcloud-1.frog-solrcloud-headless.solr-test:8983/solr\",cluster_id=\"f7122eee5d\"}","target":"http://172.23.50.53:8080/metrics","ts":"2024-05-15T08:32:26.910Z"}
{"caller":"scrape.go:1738","component":"scrape manager","level":"warn","msg":"Error on ingesting samples with different value but same timestamp","num_dropped":21,"scrape_pool":"serviceMonitor/solr-test/solr-metrics/0","target":"http://172.23.50.53:8080/metrics","ts":"2024-05-15T08:32:26.911Z"}
Disclaimer: this probably is a Solr bug rather than operator?
May 15 '24 09:05 perosb