VictoriaMetrics
Support querying label values for VictoriaLogs in Grafana
Is your feature request related to a problem? Please describe
currently, when querying logs through Grafana with VictoriaLogs, users need to manually identify and input the correct label values. in Loki, the plugin in Grafana automatically suggests available label values, reducing the need for large queries and making exploration of logs easier
Describe the solution you'd like
we would like VictoriaLogs to implement endpoints for querying label values, similar to Loki's /loki/api/v1/label/:name/values and /loki/api/v1/series endpoints. this would allow Grafana plugin to automatically suggest available label values for a given query and display log statistics without needing to run full queries over large datasets
Describe alternatives you've considered
currently, the alternative is to manually search through large datasets to find potential label values or rely on query builders. this is not ideal as it can be time-consuming and less user-friendly
Additional information
including images of how this feature works in Grafana with Loki, along with documentation links:
https://grafana.com/docs/loki/latest/reference/loki-http-api/#query-label-values https://grafana.com/docs/loki/latest/reference/loki-http-api/#query-streams
VictoriaLogs already provides the needed APIs, which can be used for obtaining label names and label values - see https://docs.victoriametrics.com/victorialogs/querying/#http-api . There are dedicated APIs, which return log stream fields / values - see https://docs.victoriametrics.com/victorialogs/querying/#querying-stream-field-names and https://docs.victoriametrics.com/victorialogs/querying/#querying-stream-field-values .
@Loori-R , @arturminchukov , make sure these APIs properly strip all the LogsQL pipes from the query passed to these APIs. Otherwise these APIs may return unexpected results.
Please do not try stripping the pipes from the query at client side (Grafana plugin or built-in web UI), since this will lead to subtle bugs without proper LogsQL parser at client side.
FYI, the commit d33efbbd952e3c462a3f7f5b3f57bd3697ddfd31 properly removes all the pipes from the query passed to HTTP querying APIs, which return log field names and values. This commit will be included in the next release.
i would really like to help VictoriaMetrics, for example can test the Grafana plugin directly from the source code (before update on grafana.com) to make sure everything works as expected, including your fixes, @valyala
if required ofc, just let me know, thanks
FYI, the commit mentioned here has been included in v1.23.3-victorialogs.
@iamhalje , thanks for the proposed help!
FYI, the commit d33efbbd952e3c462a3f7f5b3f57bd3697ddfd31 properly removes all the pipes from the query passed to HTTP querying APIs, which return log field names and values. This commit will be included in the next release.
@valyala Because of the change, the logs volume chart in Grafana now ignores all pipes when rendering. I wasn’t aware of this behavior and was misled by the graph. My initial expectation was that the logs volume chart reflects the entire query. So when I add or remove a pipe and refresh the page, I expect the chart to update accordingly. In reality, only the part of the query before the first pipe is taken into account.
My query has a pipe filter:
container:="vmselect" pod:~"vmselect-foo-.*"
| extract '"caller":<caller>'
| caller:="VictoriaMetrics/app/vmselect/main.go:282"
| extract_regexp "(?P<query>(query=.*?&))" from _msg
In this case, the logs volume chart only accounts for container:="vmselect" pod:~"vmselect-foo-.*" and and ignores the filter pipe, which can significantly affect the number of items returned.
It would be more intuitive if pipes were preserved by default when computing the chart, with an option to ignore them explicitly (e.g. ?ignore_pipes=1).
@makasim , the commit https://github.com/VictoriaMetrics/VictoriaLogs/commit/366398baf24a73954cc52e118889c56768635e04 at VictoriaLogs preserves all the pipes in the LogsQL query by default at the following HTTP endpoints, which are used by auto-completion in the Grafana plugin for VictoriaLogs and in the built-in web UI:
/select/logsql/field_names/select/logsql/field_values/select/logsql/stream_field_names/select/logsql/stream_field_values/select/logsql/streams/select/logsql/stream_ids
The commit https://github.com/VictoriaMetrics/VictoriaLogs/commit/5e1d9a8430af36c92fb5380e89d237fab42793da at VictoriaLogs takes into account pipes in the query, which do not modify the _time field, when calculating hits for the given query on the given time range. This allows displaying the expected hits chart for the query shown above.
Both commits will be included in the next release of VictoriaLogs. In the mean time it is possible to build VictoriaLogs from the commit https://github.com/VictoriaMetrics/VictoriaLogs/commit/366398baf24a73954cc52e118889c56768635e04 according to these docs and verify whether it works as expected.