Proposal: Create VictoriaLogs

[zbindenren]

Is your feature request related to a problem? Please describe. Grafana Loki is a log aggregation system inspired by Prometheus. It does not index the contents of the logs, but rather a set of labels for each log stream.

Unfortunately it is hard to run it on premises because it has dependencies to Cassandra, S3 etc...

Describe the solution you'd like It would be greate to extend vmstorage, vminsert and vmselect to support loki protocol. I think vminsert and vmselect would not be a problem, but I am not sure about vmstorage.

[faceair]

I've finished a very early version of this. https://github.com/faceair/VictoriaLogs

[syepes]

OMG! This would be just amazing and a great addition to the Victoria Family!

[faceair]

These are the changes I made:

• vmstorage:
  • change value format from float64 to []byte, use zstd compression
  • added option to fetch only timestamps, which can improve performance when query logging metrics
• vmselect:
  • extend support logql filter expression
  • support loki http api & response format
• vminsert
  • support loki push api
  • support prometheus-style data writing via tcp

I changed some code in vminsert & vmselect that was necessary, and I'm not sure how I'm going to keep it in sync with the VictoriaMetrics repository code in the future. The other bigger problem is that I have modified the storage data type of vmstorage so that I have no way to directly import the VictoriaMetrics repository and it is difficult to merge VictoriaLog into VictoriaMetrics.

Here's my new proposal https://github.com/VictoriaMetrics/VictoriaMetrics/issues/819

[zbindenren]

Cool, I wonder if this gets accepted...

I see that you @faceair already started with this 5 days ago. So we had basically the same idea 😄

[syepes]

@valyala What is your take on this functionality? Or are you guys going to focus just on the metrics?

[valyala]

What is your take on this functionality? Or are you guys going to focus just on the metrics?

We are focusing on metrics right now, but VictoriaLogs is in our plans. It can be based on https://github.com/faceair/VictoriaLogs .

[faceair]

It looks like I can maintain VictoriaLogs on my own for a while, and later I will improve the compatibility with Loki and test the real performance. I look forward to merging VictoriaLogs into VictoriaMetrics in the future.

[te4336]

@zbindenren If you happen to have the capability you can provide an S3 compatible service using Minio and then just use the Bolt-DB shipper function in the latest version of Loki. While having this as part of the Victoria Metrics family would be awesome not sure it's worth people re-inventing the wheel when there are solutions to get around some of the extraneous requirements for Loki.

[misiek08]

@te4336 almost 2 years passed and Loki is still unusable for scenarios with real logging data. Trying to query anything hitting more than 50k logs is exploding servers :)

I'll try to contribute to VictoriaLogs and I'm glad I found this issue and VictoriaLogs repo ❤️

[mxssl]

Do want!

[v1gnesh]

Hello,

Post the recent community day and expressed interest to create VictoriaLogs, please see if there are patterns & methods from this that can be rebuilt in Go, for VL.

https://www.youtube.com/watch?v=23iHoDeO7M4 https://www.uber.com/en-IN/blog/reducing-logging-cost-by-two-orders-of-magnitude-using-clp/

[tenmozes]

We started working on Victoria Logs, you can find some information here https://www.youtube.com/live/Gu96Fj2l7ls?feature=share&t=1952

[zbindenren]

We started working on Victoria Logs, you can find some information here https://www.youtube.com/live/Gu96Fj2l7ls?feature=share&t=1952

Hope that there will be a grafana datasource. Looks good.

[techwavedev]

looking forward to see a datasource for grafana or to use it as Loki one, and the possibility to use victoriametrics cluster that already exists

[valyala]

Good news - the initial release of VictoriaLogs is ready for use - see these docs.

VictoriaLogs accepts logs from Filebeat, Fluentbit, Logstash and Vector - see these docs. Promtail and Fluentd support is on the way.

VictoriaLogs provides user-friendly query language - LogsQL. See these docs

VictoriaLogs integrates with existing command-line tools such as less, tail, grep, jq, awk, head, etc . See these docs

[arrowcircle]

Hey! Great work! Is it possible to access these logs from grafana?

[StefanSa]

Hi there, Also very interested in an Integartion with Grafana

[davidsome]

hi，does VictoriaLogs has cluster version？

[denisgolius]

hi，does VictoriaLogs has cluster version？

Hello, not for now, but we have it in our roadmap - see https://docs.victoriametrics.com/VictoriaLogs/Roadmap.html

[denisgolius]

looking forward to see a datasource for grafana or to use it as Loki one, and the possibility to use victoriametrics cluster that already exists

FYI: starting from v0.3.0 VictoriaLogs support data ingestion via Promtail (aka default log shipper for Grafana Loki)

[zbindenren]

hi，does VictoriaLogs has cluster version？

Hello,

not for now, but we have it in our roadmap - see https://docs.victoriametrics.com/VictoriaLogs/Roadmap.html

Is grafana datasource not on the roadmap anymore?

[valyala]

Is grafana datasource not on the roadmap anymore?

Grafana datasource for VictoriaLogs will be added eventually. There are the following options exist:

1. To add a new datasource to Grafana, which is specifically optimized for VictoriaLogs. There are high chances that Grafana won't include this datasource to standard Grafana installation, so it should be installed by users via inconvenient hacks. See the example with VictoriaMetrics datasource for Grafana. We were trying to include this datasource into standard Grafana installation, but it wasn't included. They proposed yearly subscription for including VictoriaMetrics datasource into standard Grafana installation, but the price was too high for us.

2. To implement an API at VictoriaLogs for one of the supported datasource for logs in Grafana. For example, Loki or Elasticsearch. Then these existing datasources could be used for querying VictoriaLogs. This solution may prevent to use LogsQL if these datasources parse or validate queries before sending them to Loki / Elasticsearch. In this case we need to add support for Loki query language or for Elasticsearch query language into VictoriaLogs. This isn't a good solution, since LogsQL is the essential part of VictoriaLogs, which simplifies writing typical queries for logs.

There is a third option - to evolve native web UI included in VictoriaLogs, in order to make it useful for most cases with log analysis and debugging.

[lddlww]

why victoriaLogs can't support Loki query like victoriaMetrics support promql？

[helmut72]

Is web ui planned to get (also that much free) features like Grafana?

[QuentinN42]

Hello, I don't see in the roadmap if it will be compatible with grafana (as a Loki datasource), I currently get this :

Is someone working on making VL Loki compatible ?

EDIT: will be happy to contribute :)

[JakeCooper]

We L O V E VictoriaMetrics at Railway

Currently looking to setup a Grafana -> LGTM stack and don't want to host Loki

Would use this in a heartbeat if it spoke Loki for querying through Grafana 😍 . Happy to contribute/sponsor/etc

[valyala]

why victoriaLogs can't support Loki query like victoriaMetrics support promql？

It shouldn't be hard from technical perspective adding support for Loki query language into VictoriaLogs. Probably, it will be added in the future. But I personally don't like this direction, since Loki query language - LogQL - is awful query language for logs from usability and readability perspective comparing to LogsQL - query language for VictoriaLogs. A few practical examples:

1. Select all the log entries with the error word in the log message:

   • Loki: in general, it is impossible - you need specifying log stream selector, which matches all the log streams stored in Loki. You need to know a common label, which exists across all the streams in order to construct such a selector. For example, if you know that all the log streams contain host label, then the log stream selector, which selects all the streams, may be expressed as {host=~".+"}. In this case the query, which selects all the log lines with the error word will look like {host=~".+"} |= "error" - it uses |= line filter. But this query can return only a part of matching log entries, because Loki limits the number of results a single query can return. The limit is set to 5000 by default - see max_entries_limit_per_query config option here. Of course, you can set bigger limit, but this may result in increased RAM usage at Grafana Loki.
   • VictoriaLogs: just type error word in the query - that's it! You can try executing the following command on the host where VictoriaLogs runs in order to test how does it work:

   curl http://localhost:9428/select/logsql/query -d 'query=error'
   

   This query returns all the log entries from all the log streams, which contain the error word in the log message. If VictoriaLogs contains billions of log entries with the error word inside log message, then all these billions of log entries will be returned in query results - there are no any limits on the number of returned matching log entries at VictoriaLogs side. Such queries do not crash VictoriaLogs and do not use additional RAM, CPU and disk space, because VictoriaLogs performs the query and returns query results in streaming manner, e.g. it reduces query execution speed when the client reads query response slowly. It pauses query execution when the client pauses reading query results. It cancels the query and frees up all the occupied resources when the client closes response connection. This allows processing arbitrary number of matching log entries at client side in command-line mode via Unix pipes. See these docs for details.

2. Select all the logs, which contain the error word written in any case. For example, error, Error, ERROR, eRrOr, etc.

   • Loki: {host=~".+"} |~ "(i)error" . Loki forces using non-trivial regexp in this case, which is hard to write, read and maintain. It also may slow down the query significantly comparing to a specialized syntax.
   • VictoriaLogs: i("error") . See case-insensitive filters in LogsQL.

3. Select all the logs, which contain either error or warn words:

   • Loki: {host=~".+"} |~ "error|warn" . Loki forces using regular expression for such a simple query, since it has no syntax, which allows selecting logs with the listed words. Regular expression are non-trivial to write, read and maintain. They also may execute much slower comparing to simple word filters.
   • VictoriaLogs: error OR warn . The OR operator is used here - see these docs for details.

4. Select all the logs with the following list of errors / phrases: error, auth failed and invalid password:

   • Loki: {host=~".+"} |= "error" |= "auth failed" |= "invalid password" . This |= syntax look very unnatural in the query.
   • VictoriaLogs: error "auth failed" "invalid password" or error AND "auth failed" AND "invalid password" . The AND operator is used here - see these docs for details.

5. Select all the logs with the error word, while dropping logs which contain simultaneously auth failed and invalid password phrases. For example, the message error: auth failed: backend error should match, since it doesn't contain "invalid password" phrase, while the message error: auth failed: invalid password should be filtered out, since it contains both auth failed and invalid password phrases:

   • Loki: this query cannot be expressed in Loki :(
   • VictoriaLogs: error !("auth failed" AND "invalid password") . The ! operator (aka NOT) and parenthesis are used here. See these docs for details.

6. Select all the logs with the error word followed by "auth failed", while skipping logs where "auth failed" goes in front of error word:

   • Loki: {host=~".+} |~ "error.+auth failed" . Again, Loki forces using non-trivial regular expressions for such a simple query.
   • VictoriaLogs: seq("error", "auth failed") . The sequence filter is used here.

On top of this, contrary to Grafana Loki, VictoriaLogs supports efficient ingesting and querying high-cardinality labels (aka log fields ) such as trace_id, ip, user_id, etc., with a simple syntax: field_name:query_here. For example, the following VictoriaLogs query returns logs where the trace_id label contains ABCD-EFGH string and the ip field is in the range [127.0.0.0 - 127.0.0.255]:

trace_id:"ABCD-EFGH" ip:ipv4_range("127.0.0.0/24")

Grafana Loki doesn't support high-cardinality labels, so it doesn't provide query syntax for selecting on such labels.

This makes dubious implementing Loki query language support in VictoriaLogs.

[valyala]

Is web ui planned to get (also that much free) features like Grafana?

We are going to evolve web UI for VictoriaLogs in the direction of better usability. TBH, Grafana UI for logs isn't the best example of useful UI for logs. VictoriaLogs will have much better UI from usability perspective than Grafana over time.

[valyala]

Is someone working on making VL Loki compatible ?

There are some chances to add support for Loki query language in VictoriaLogs - see this answer for more details.

There are higher chances to add support for essential Loki-compatible querying APIs into VictoriaLogs, such as /api/v1/labels and /api/v1/label/.../values, which can be useful for labels' auto-completion during query construction in web UI provided by VictoriaLogs.

[helmut72]

Is web ui planned to get (also that much free) features like Grafana?

We are going to evolve web UI for VictoriaLogs in the direction of better usability. TBH, Grafana UI for logs isn't the best example of useful UI for logs. VictoriaLogs will have much better UI from usability perspective than Grafana over time.

Thanks for answering. Yes, of course it's not a good UI for logs, but can be used for more than logs and does have free multiuser support with different auth flavors like OIDC, LDAP. Also the builder in explorer view helped a bit getting used to.