prometheus icon indicating copy to clipboard operation
prometheus copied to clipboard

Published 20 hours ago • verified publisher icon prometheus

'num of chunks' shown as a negative in web-interface

Open mator opened this issue 2 years ago • 9 comments

What did you do?

Looking at my prom web-interface http://127.0.0.1:9090/tsdb-status Number of Chunks is shown as -456475

$ curl -s localhost:9090/api/v1/status/tsdb | jq ".data.headStats"
{
  "numSeries": 1369489,
  "numLabelPairs": 74540,
  "chunkCount": -456479,
  "minTime": 1655618400121,
  "maxTime": 1655624577252
}

here's overview of my tsdb:

data# promtool tsdb list .
BLOCK ULID                  MIN TIME       MAX TIME       DURATION       NUM SAMPLES  NUM CHUNKS   NUM SERIES   SIZE
01G3N2603Y0PE40CKX6VRXSAB2  1652983200122  1653177600000  53h59m59.878s  471451639    3929643      38725        433448110
01G3TVJM9Q7DH1G3W2PAQGCR55  1653177600122  1653372000000  53h59m59.878s  457038293    3809713      40159        428479489
01G40MZEYGGEWVBMG9S1WN1CYX  1653372000122  1653566400000  53h59m59.878s  922604697    7740319      124242       671688701
01G46EC4WJKF96NBZRSK3KPSR8  1653566400122  1653760800000  53h59m59.878s  1592657470   13524550     300887       1096184758
01G4C7RPZE6GZHAFB0TTDW23FR  1653760800122  1653955200000  53h59m59.878s  1646218783   13867490     260968       1092629725
01G4J15ANBSWCQ4Y1M7F60WAY9  1653955200122  1654149600000  53h59m59.878s  1616420954   13467860     131754       1031044925
01G4QTHY9JFFRDGEESHS64R1SW  1654149600122  1654344000000  53h59m59.878s  1597206744   13310693     127249       1020519479
01G4XKYJ5S95FTYG8PHZSJW9FV  1654344000122  1654538400000  53h59m59.878s  1593493983   13275555     124950       1015372665
01G53DEHFQBH3C2VNKF7DZVMQJ  1654538400122  1654732800000  53h59m59.878s  2068417186   24038886     7649212      2497952525
01G596S0PH1X40106FMMNJFBJX  1654732800122  1654927200000  53h59m59.878s  4084036508   70158281     34171232     8219917184
01G5F0ASCEEEJ0FZKV1GQT0ES0  1654927200122  1655121600000  53h59m59.878s  4076499715   67967636     32992100     7979406464
01G5MJPFBSCXFHZQ0YZ1X5F7Y5  1655121600000  1655316000000  54h0m0s        4081998687   68999158     33334540     8089027588
01G5PQD59BKGWNGJFRN9JFE21Y  1655316000122  1655380800000  17h59m59.878s  1360672087   22508451     11747030     2740127438
01G5RN6RFPHWX35MDW8JDPW499  1655380800122  1655445600000  17h59m59.878s  1354870158   23113953     12500476     2853552095
01G5TC4CRENP7G2NA0J70996Q7  1655445600000  1655510400000  18h0m0s        1305378542   22633013     12196284     2789717520
01G5WGRG2Z2VAWABR12M1ADEKE  1655510400121  1655575200000  17h59m59.879s  1349972214   22821238     12005950     2760530729
01G5WYG4WJ87JK79PEGERQQQSQ  1655575200000  1655596800000  6h0m0s         449694097    7721930      4760195      1011026199
01G5WYE90ZAP5YNEVNDXCRSWST  1655596800000  1655604000000  2h0m0s         149984245    2479712      1819318      356725823
01G5X5A0SJCCZ9920YG2DES4ZD  1655604000121  1655611200000  1h59m59.879s   149806749    2518409      1865448      366250244
01G5XC5R1J3305D6C37ZXZ3GX4  1655611200121  1655618400000  1h59m59.879s   149941492    2537842      1885470      369396192

and sum of chunks:

data# promtool tsdb list . | sed '1d' | awk '{ s+=$6 } END {print s}'
420424332

What did you expect to see?

expecting num of chunks (sum) to be positive value

What did you see instead? Under which circumstances?

.

System information

Linux 3.10.0-1160.15.2.el7.x86_64 x86_64

Prometheus version

prometheus, version 2.36.0 (branch: HEAD, revision: d48f381d9a4e68c83283ce5233844807dfdc5ba5)
  build user:       root@b3126bd1c115
  build date:       20220530-13:56:56
  go version:       go1.18.2
  platform:         linux/amd64

Prometheus configuration file

.

Alertmanager version

.

Alertmanager configuration file

.

Logs

.

mator avatar Jun 19 '22 07:06 mator

restarted prometheus -> can't reproduce anymore...

data# curl -s localhost:9090/api/v1/status/tsdb  | jq '.data.headStats'
{
  "numSeries": 1369495,
  "numLabelPairs": 74540,
  "chunkCount": 813680,
  "minTime": 1655618400000,
  "maxTime": 1655625838444
}

mator avatar Jun 19 '22 08:06 mator

just checked, and it is negative again:

$ systemctl status prometheus
● prometheus.service - The Prometheus monitoring system and time series database.
   Loaded: loaded (/usr/lib/systemd/system/prometheus.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2022-06-19 11:01:40 MSK; 2 days ago
     Docs: https://prometheus.io
 Main PID: 7205 (prometheus)
   CGroup: /system.slice/prometheus.service
           └─7205 /usr/bin/prometheus --config.file=/etc/prometheus/prometheus.yml --storage.tsdb.path=/opt/prometheus/data --storage.tsdb.retention.time=45d --enable-fea...
...

$ curl -s  localhost:9090/api/v1/status/tsdb  | jq '.data.headStats'
{
  "numSeries": 125051,
  "numLabelPairs": 8998,
  "chunkCount": -757793,
  "minTime": 1655820000122,
  "maxTime": 1655828411870
}


data# promtool tsdb list .
BLOCK ULID                  MIN TIME       MAX TIME       DURATION       NUM SAMPLES  NUM CHUNKS   NUM SERIES   SIZE
01G3N2603Y0PE40CKX6VRXSAB2  1652983200122  1653177600000  53h59m59.878s  471451639    3929643      38725        433448110
01G3TVJM9Q7DH1G3W2PAQGCR55  1653177600122  1653372000000  53h59m59.878s  457038293    3809713      40159        428479489
01G40MZEYGGEWVBMG9S1WN1CYX  1653372000122  1653566400000  53h59m59.878s  922604697    7740319      124242       671688701
01G46EC4WJKF96NBZRSK3KPSR8  1653566400122  1653760800000  53h59m59.878s  1592657470   13524550     300887       1096184758
01G4C7RPZE6GZHAFB0TTDW23FR  1653760800122  1653955200000  53h59m59.878s  1646218783   13867490     260968       1092629725
01G4J15ANBSWCQ4Y1M7F60WAY9  1653955200122  1654149600000  53h59m59.878s  1616420954   13467860     131754       1031044925
01G4QTHY9JFFRDGEESHS64R1SW  1654149600122  1654344000000  53h59m59.878s  1597206744   13310693     127249       1020519479
01G4XKYJ5S95FTYG8PHZSJW9FV  1654344000122  1654538400000  53h59m59.878s  1593493983   13275555     124950       1015372665
01G53DEHFQBH3C2VNKF7DZVMQJ  1654538400122  1654732800000  53h59m59.878s  2068417186   24038886     7649212      2497952525
01G596S0PH1X40106FMMNJFBJX  1654732800122  1654927200000  53h59m59.878s  4084036508   70158281     34171232     8219917184
01G5F0ASCEEEJ0FZKV1GQT0ES0  1654927200122  1655121600000  53h59m59.878s  4076499715   67967636     32992100     7979406464
01G5MJPFBSCXFHZQ0YZ1X5F7Y5  1655121600000  1655316000000  54h0m0s        4081998687   68999158     33334540     8089027588
01G5PQD59BKGWNGJFRN9JFE21Y  1655316000122  1655380800000  17h59m59.878s  1360672087   22508451     11747030     2740127438
01G5RN6RFPHWX35MDW8JDPW499  1655380800122  1655445600000  17h59m59.878s  1354870158   23113953     12500476     2853552095
01G5TC4CRENP7G2NA0J70996Q7  1655445600000  1655510400000  18h0m0s        1305378542   22633013     12196284     2789717520
01G5WGRG2Z2VAWABR12M1ADEKE  1655510400121  1655575200000  17h59m59.879s  1349972214   22821238     12005950     2760530747
01G5Y7NBHP4FAN3BQYABD7DWF2  1655575200000  1655640000000  18h0m0s        1135941873   18301054     9514081      2214333119
01G60CA41SXRZM10X4X43RDYBC  1655640000122  1655704800000  17h59m59.878s  531838024    4428977      123724       348106671
01G62A3NC7ZYX1K4DMP8GKFMPZ  1655704800122  1655769600000  17h59m59.878s  532831464    4438959      124862       348021261
01G62YPRN7NQEP2BRXM92YV11C  1655769600122  1655791200000  5h59m59.878s   177472401    1478802      124090       123954252
01G62QTX9K9NRMKZ7Y6SZB5PWR  1655791200122  1655798400000  1h59m59.878s   59231830     493735       123901       48652085
01G62YPMHMN7B3HJJ8SZ8AMG3J  1655798400122  1655805600000  1h59m59.878s   59166027     497528       125224       53388582
01G635JBSJ21A9R02YBE4GM6XC  1655805600122  1655812800000  1h59m59.878s   59545159     496838       124998       49054791
01G63CE31KBMFX3CAVXB0MFKPS  1655812800122  1655820000000  1h59m59.878s   59718392     497152       125111       49034699

can someone look please?

mator avatar Jun 21 '22 16:06 mator

I got the same issue with prometheus 2.37.0

prom-so-many-chunks

paulfantom avatar Aug 16 '22 19:08 paulfantom

That's weird. It seems to be based of this metric: prometheus_tsdb_head_chunks, which I assume is going negative too?

LeviHarrison avatar Aug 28 '22 12:08 LeviHarrison

promtool tsdb list is for the DB. The negative value is for the HEAD. It is something worth investigating.

Can we get the values for

prometheus_tsdb_head_chunks_created_total prometheus_tsdb_head_chunks_removed_total prometheus_tsdb_head_chunks

Thanks

roidelapluie avatar Sep 05 '22 10:09 roidelapluie

Hi! Observed the same issue in Prometheus 2.45.0:

curl -s localhost:9090/api/v1/status/tsdb | jq '.data.headStats' { "numSeries": 1752605, "numLabelPairs": 27412, "chunkCount": -14343, "minTime": 1704801600055, "maxTime": 1704808414246 } Related metrics: `prometheus_tsdb_head_chunks -14339

prometheus_tsdb_head_chunks_created_total 1.163943301e+09

prometheus_tsdb_head_chunks_removed_total 1.16395764e+09 ` Regards.

jumakasy avatar Jan 09 '24 14:01 jumakasy