Slow inserts remain high, although the churn rate and the number of active series have decreased

[escapekyg]

Is your question request related to a specific component?

vmstorage, slow insert

Describe the question in detail

Previously, our vmstorage experienced high levels of churn rate, active timeseries, and slow inserts, which often led to out-of-memory (OOM) issues. Recently, we did the following to improve: 1.upgraded our machine's memory from 64GB to 128GB 2.stopped ingesting some metrics with frequently changing label values. 3.added commanline flag -storage.maxDailySeries=1500000

Now, we observed the cardinality explorer, we have many series, as well as labels with a high number of values that are indeed necessary to ingest. Puzzlingly, although the churn rate and active timeseries have decreased, the slow inserts remain high. Is there a way to more accurately identify which series are causing the slow inserts to remain high, or if there are other underlying reasons? Thank you!

we use version 1.100.1 we deploy a cluster with 3 machines, each machine has vminsert/vmselect/vmstorage, commanline for each machine: ./vmstorage -storageDataPath=/data/victoriaMetrics -httpListenAddr=10.0.0.1:18000 -vminsertAddr=10.0.0.1:18001 -vmselectAddr=10.0.0.1:18002 -storage.cacheSizeStorageTSID=1GB -storage.cacheSizeIndexDBDataBlocks=1GB -storage.cacheSizeIndexDBIndexBlocks=1GB -bigMergeConcurrency=8 -storage.maxDailySeries=1500000 -search.maxUniqueTimeseries=2000000 -retentionPeriod=1y ./vminsert -replicationFactor=2 -httpListenAddr=10.0.0.1:18004 -storageNode=10.0.0.1,10.0.0.2,10.0.0.3 ./vmselect -replicationFactor=2 -cacheDataPath=/vm/tmp -httpListenAddr=10.0.0.1:18003 -dedup.minScrapInterval=15s -storageNode=10.0.0.1,10.0.0.2,10.0.0.3

Troubleshooting docs

• [x] General - https://docs.victoriametrics.com/troubleshooting/
• [x] vmagent - https://docs.victoriametrics.com/vmagent/#troubleshooting
• [x] vmalert - https://docs.victoriametrics.com/vmalert/#troubleshooting

[hagen1778]

Hello @escapekyg!

Puzzlingly, although the churn rate and active timeseries have decreased, the slow inserts remain high

Slow inserts aren't always a sign of churn rate, it could be a sign of insufficient tsid cache size. See also SlowInserts panel description:

The percentage of slow inserts comparing to total insertion rate during the last 5 minutes. The less value is better. If percentage remains high (>10%) during extended periods of time, then it is likely more RAM is needed for optimal handling of the current number of active time series. In general, VictoriaMetrics requires ~1KB or RAM per active time series, so it should be easy calculating the required amounts of RAM for the current workload according to capacity planning docs. But the resulting number may be far from the real number because the required amounts of memory depends on many other factors such as the number of labels per time series and the length of label values. See also this issue for details.

Is there a way to more accurately identify which series are causing the slow inserts to remain high

New series can be detected via the following flag:

-logNewSeries Whether to log new series. This option is for debug purposes only. It can lead to performance issues when big number of new series are ingested into VictoriaMetrics

Or by comparing days in Cardinality explorer (the green number shows change of series for a metric name comparing to prev day):

1.upgraded our machine's memory from 64GB to 128GB

As I said in previous issue, I find it very disturbing that you need 3 vmstorages with 128GiB of mem each to handle 1Mil of Active Time series. Normally, this won't require more than 10GiB in total. I'd start investigating why the memory usage is so high. I remember you shared memory profile in previous issue, but I didn't have time to check it yet. I'll let you know once I have any updates.

[hagen1778]

@escapekyg could you please provide screenshots of the following panels from VictoriaMetrics official Grafana dashboard for cluster version:

1. vmstorage CPU and memory usage
2. vmstorage Active merges and Merge speed
3. vmstorage Disk space usage % and usage by type %
4. vmstorage LSM parts by type

Thanks!

[escapekyg]

Hello @hagen1778 ! Thanks for reply !

vmstorage CPU and memory usage vmstorage Active merges and Merge speed vmstorage Disk space usage % and usage by type % vmstorage LSM parts by type

The following are the respective screenshots of one node of the cluster, along with other important panels. After we added -storage.maxDailySeries=1500000, the size of the indexdb file seems to have stopped increasing; it has stayed at 30GB in recent days.

[hagen1778]

I am super confused. In your prev issue you attached memory profile. The profile shows that 30GiB of mem was used during reading the metaindex.bin file. This file contains very little info, just information about the data part (max/min timestamp, number of elements, etc). I've checked size of this file in other installations. Specifically, one vmstorage node with 2.4TiB of data on disk, 2.5Mil Active time series, 6GiB RAM used. The total size of all metaindex.bin files on this node is below 15MiB. When asking about number of parts screenshot, I suspected you could be having too many unmerged parts (each part has metaindex file) on disk, but it is not true. I don't understand why you need 60GiB of mem for 800K Active time series...

Actions to take:

1. Would be great if you could try writing the same info to the new VM installation, just for test. You could even try running VM single, for your workload it seems like one VM single with 4vCPU and 16GiB of mem will do.
2. If the new installation will show the same level of mem usage - please, show me the example data you write into it.

[perederyaev]

Hi, this looks similar to my issue https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6198 . When VM gets new timeseries with null values - it doesn't insert them and shows them like slow inserts and cache misses.

[hagen1778]

@escapekyg can I ask you to execute the following cmd for one of the vmstorage data folders?

ls -hs vmstorage-data/indexdb/*/* | grep metaindex.bin

[escapekyg]

Hello @hagen1778 ! Thanks for your reply again!

Would be great if you could try writing the same info to the new VM installation, just for test. You could even try running VM single, for your workload it seems like one VM single with 4vCPU and 16GiB of mem will do. If the new installation will show the same level of mem usage - please, show me the example data you write into it.

The data is in our customer's environment, we can't frequently access it, and it's hard to simulate the data like customer's.

can I ask you to execute the following cmd for one of the vmstorage data folders?

metaindex.txt The cmd result is in attachment file, thank you!

[f41gh7]

I think an issue related to the design of metaindex. In case, when index records have large entities ( > 20k bytes). VictoriaMetrics may consume extra memory for storing metaindex.

Usually, the size of metrics are relatively small. But if there are a lot of metrics with huge label values size, it increases chance of bloat metaindex size.

it also give a chance, that during indexdb merge process it ll require x2 memory and cause immediate OOM.

[hagen1778]

In other words, you experience so high mem usage because of labels with big size. In order to reduce memory usage, one shouldn't have very long labels. If you avoid having that, the memory usage will drop significantly for your cluster. The best way would be to get rid of time series with too wide labels and re-ingest data again in a fresh cluster to cleanup the indexes.

We'll try to patch VM to avoid this problem in future. Maybe by limiting the max label size, or by re-thinking the metaindex.bin structure. But the fastest way to mitigate the issue would be as described above.

[escapekyg]

@hagen1778 @f41gh7 Thank you! It's collecting metrics from a container environment, the labels, such as pod, id, name, image, etc., are quite long. Also, I am still trying to figure out why the rate of slow inserts remains high. I added the -logNewSeries=true flag to the command line, but the number of new series over 15 minutes is only several hundred, significantly lower than what vm_slow_row_inserts_total indicates. Could this issue be related to the excessively long labels? Is it possible to increase the memory allocated to the TSID index to reduce cache misses?

[hagen1778]

It's collecting metrics from a container environment, the labels, such as pod, id, name, image, etc., are quite long.

Could you provide an example of too long labels, because what you say is normal labels. The labels that could be a concern are more than 1KB size or if one time series has many labels of that size.

[hagen1778]

Also, I am still trying to figure out why the rate of slow inserts remains high. I added the -logNewSeries=true flag to the command line, but the number of new series over 15 minutes is only several hundred, significantly lower than what vm_slow_row_inserts_total indicates.

This could be related to this comment https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6176#issuecomment-2080069779

Do you collect metrics via vmagent?

[escapekyg]

@hagen1778

Could you provide an example of too long labels, because what you say is normal labels. The labels that could be a concern are more than 1KB size or if one time series has many labels of that size.

I check some of them, I think it's possbile to have some metrics which are over 1kb, for example, the lable value of image is sha256 hash value, and they have multi labels with long length

This could be related to this comment https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6176#issuecomment-2080069779

ok, I will check this next week in customer's environment.

Do you collect metrics via vmagent?

No, we don't

[escapekyg]

@hagen1778 @perederyaev is there any way to find out the specific series cause slow inserts?

[escapekyg]

Also, I am still trying to figure out why the rate of slow inserts remains high. I added the -logNewSeries=true flag to the command line, but the number of new series over 15 minutes is only several hundred, significantly lower than what vm_slow_row_inserts_total indicates.

This could be related to this comment #6176 (comment)

Do you collect metrics via vmagent?

Hello @hagen1778 ! Today, I checked the metric data and found that there were no metrics with null values. I used -logNewSeries to log the new series, but they represent a very small percentage of the total number of inserts. But the slow inserts is still above 30%, could it be possible that many inserts miss the TSID cache?

Now the machines still have sufficient free memory, I haven't configured -storage.cacheSizeStorageTSID, so theoretically, there should be no limit on the TSID cache size?

[f41gh7]

Recommended settings to minimize memory footprint:

-maxLabelValueLen=512
-maxLabelsPerTimeseries=30

After applying changes, it requires full indexdb rotation to decrease memory usage. Basically it's a value of retentionPeriod flag.

Current indexdb architecture stores all sparse indexes in-memory. It may use a lot of memory, if metrics have huge label values. Potentially, it could be solved by migration to the partitioned indexdb (monthly,daily). But there is no such plans for now.

[escapekyg]

Recommended settings to minimize memory footprint:

-maxLabelValueLen=512
-maxLabelsPerTimeseries=30

After applying changes, it requires full indexdb rotation to decrease memory usage. Basically it's a value of retentionPeriod flag.

Current indexdb architecture stores all sparse indexes in-memory. It may use a lot of memory, if metrics have huge label values. Potentially, it could be solved by migration to the partitioned indexdb (monthly,daily). But there is no such plans for now.

Thanks @f41gh7 ! We will consider your advice. But now the memory is sufficient, and the churn rate is low, but the slow inserts is still above 30%, is this also related to the long label?

[f41gh7]

Recommended settings to minimize memory footprint:

-maxLabelValueLen=512
-maxLabelsPerTimeseries=30

After applying changes, it requires full indexdb rotation to decrease memory usage. Basically it's a value of retentionPeriod flag. Current indexdb architecture stores all sparse indexes in-memory. It may use a lot of memory, if metrics have huge label values. Potentially, it could be solved by migration to the partitioned indexdb (monthly,daily). But there is no such plans for now.

Thanks @f41gh7 ! We will consider your advice. But now the memory is sufficient, and the churn rate is low, but the slow inserts is still above 30%, is this also related to the long label?

Regarding slow inserts. You changed value for -storage.cacheSizeStorageTSID=1GB. It may trigger slow insert, if current size of TSID cache isn't enough for your ingestion rate. I'd recommend to use default value or adjust it for your needs. Check Troubleshooting grafana panel for insights.

Also, unsorted labels may trigger slow insert. It could happen, if client doesn't follow requirement for metric sorting. You can enforce sorting with flag sortLabels at vminsert.

[escapekyg]

Recommended settings to minimize memory footprint:

-maxLabelValueLen=512
-maxLabelsPerTimeseries=30

After applying changes, it requires full indexdb rotation to decrease memory usage. Basically it's a value of retentionPeriod flag. Current indexdb architecture stores all sparse indexes in-memory. It may use a lot of memory, if metrics have huge label values. Potentially, it could be solved by migration to the partitioned indexdb (monthly,daily). But there is no such plans for now.

Thanks @f41gh7 ! We will consider your advice. But now the memory is sufficient, and the churn rate is low, but the slow inserts is still above 30%, is this also related to the long label?

Regarding slow inserts. You changed value for -storage.cacheSizeStorageTSID=1GB. It may trigger slow insert, if current size of TSID cache isn't enough for your ingestion rate. I'd recommend to use default value or adjust it for your needs. Check Troubleshooting grafana panel for insights.

Also, unsorted labels may trigger slow insert. It could happen, if client doesn't follow requirement for metric sorting. You can enforce sorting with flag sortLabels at vminsert.

Thank you! We had already removed -storage.cacheSizeStorageTSID before, but it didn't work. Now we use sortLabels, and the slow inserts have significantly decreased.

[valyala]

FYI, the default value for -maxLabelValueLen command-line flag has been reduced from 16KB to 1KB starting from v1.102.0-rc1 release.

[valyala]

FYI, the default value for -maxLabelValueLen command-line flag has been increased from 1KiB to 4KiB in the commit e3d5714f6fd4e0eb7767258b3d8deffa1dda82fc , since it has been appeared that Kubernetes monitoring can generate labels longer than 1KiB by default.