VictoriaLogs
VictoriaLogs copied to clipboard
faceair
update vmcluster to 1.65.0
fixed issues:
- memory leak in vmselect
- broken labels api
- corrupted logs within vmstorage
- wrong direction in vmselect
- dead connections in connPool
- duplicated fetchDataOption resulted empty func result
- unexpected limit result
https://github.com/faceair/VictoriaLogs/pull/7/commits/c287aab9681a7ad6800842510439d96beb6b4076 can revert with next vm release, since upstream has fixed, see https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1597
Upgrading the dependency version is probably only the first step, we also need to follow up the storage module from the fork to the latest version, which will be more troublesome. I can merge this PR first.
I think we shouldn't change value type in vmstorage, how about extend vmstorage with a new field data to store log data, each log's value defaults to 1, so we could benefit with most MetricsQL funcs. Besides, we could support exemplars with this extended data field.
WDYT?
Great, maybe we can just archive this project, and restart the extend work with original VM to get ride of the follow up things.
Further changes should just depends on the extended data field, so we can easily rebese from upstream
It feels good to just open a new branch to do this, and when we're done change the current master branch to something else.
So it looks like we don't need to merge this pr for now, so let's just develop it on the new branch.
Also I think git rebase might not work very well. Because we delete all the modules except for the storage module. We might need a script to help us diff and patch the latest storage code periodically.
IMO, we can just keep vm in sync with upstream, if we want this gets merged in the future.
With extended data field, we can do more than just metrics, logs, exemplars, traces, maybe even more, profiling system like pyroscope.
Different data has different storage and query requirements, and there are many differences in storage design. loki is a special case, its design deliberately leaned towards metrics, it is not quite the same as traditional logging systems. A single store can hold all the observations, but this is certainly not the optimal choice.
But as far as supporting only loki is concerned, we can really try.
Or just consider implementing https://github.com/VictoriaMetrics/VictoriaMetrics/issues/819 , and prepare a more complete proposal to submit to VM, and we can experiment with this repository first.
OK, I need some time to familiar with current storage module, and today I have to deal with another bug in VL, unexpected result with query like count_over_time(({k="v"}|~"sth")[1m]).