VictoriaMetrics
Error while inserting data more then 2 days in future.
Describe the bug
Writing data with future day, produces:
2020-10-11T22:52:07.001Z error VictoriaMetrics/lib/storage/storage.go:1237 warn occurred during rows addition: cannot insert row with too big timestamp 1893452400000 exceeding the current time; maximum allowd timestamp is 1602629526000; propbably you need updating -retentionPeriod command-line flag
To Reproduce Insert data with a future day (more then +2 days from current date)
Expected behavior Write should be successfull
Screenshots
- not needed
Version
$ ./victoria-metrics-prod --version
victoria-metrics-20201006-113018-tags-v1.43.0-0-g2a6fa5395
Used command-line flags
./victoria-metrics-prod -maxConcurrentInserts 1024 -memory.allowedPercent=40 -retentionPeriod=1200
Additional context The line of code says it clearly: https://github.com/VictoriaMetrics/VictoriaMetrics/blob/1f33dd717f2526462807f9337e76f54060b3031d/lib/storage/table.go#L380
This should be a bug, it is not uncommon to store data in the future, for example meteo weather forecast, those will be deleted and re-added after a new forecast is made, but that is normal.
Please disable this limitation I need to store future data for my forecasts too
Hi. Can you find time for this issue? It's very important for me too. We using your database because it very fast instead of Thanos and Prometheus.
VictoriaMetrics architecture has been designed with the assumption that timestamps for the ingested samples shouldn't go too far in the future. That's why there could be unexpected issues after removing this limitation. This needs additional investigation before allowing storing samples with timestamps from the future.
@valyala So, if I understood correct for now we can use at least the future timestamps till +2 days, right ? Does it make sense for the latest VM versions ?
well, I think I found the answer after checking the code:
https://github.com/VictoriaMetrics/VictoriaMetrics/blob/cluster/lib/storage/table.go#L380
VictoriaMetrics architecture has been designed with the assumption that timestamps for the ingested samples shouldn't go too far in the future. That's why there could be unexpected issues after removing this limitation. This needs additional investigation before allowing storing samples with timestamps from the future.
Do you have any more intuition as what "unexpected issues" might occur, or does even figuring that out involve a deep dive investigation? I'm looking to store forecast data for up to 30 days in the future, and am currently recompiling VM with the above mentioned line modified.
@tedpearson Curious if you have this working. I tweaked it to a hardcoded value of 7 days (which meets my needs, but curious if you are using it with success with a longer duration?
@ober37 sorry for not getting back to you until now. I've had zero issues with my +30 day recompilation.
Update for this issue:
For now, it’s not safe to allow ingestion metrics with timestamps bigger then 2 days. It’ll lead to incorrect metrics registration at the index and metrics will be partly available or unavailable for search after indexdb rotation.
Hey hey, is this still an unsolvable issue? Are there any updates?
Thanks in advance
One stupid question: what could happen if we trick our VM server to be a few years in the future? As long as we don't have a retention time, it should be fine? That way we could hack this? 😅
We're working with weather forecast and budget data as well - which is usually up to one year in the future. From my understanding of @f41gh7's analysis, just bumping the maxTimestamp variable to 365 or more instead of 2 days would cause pretty significant issues. Is @namirsab's approach maybe more effective, albeit technically maybe not super "clean"?
