LogQL: Selective label extraction with logfmt pipeline

[dannykopping]

Is your feature request related to a problem? Please describe. When processing log lines through the logfmt pipeline, I would like to be able to selectively extract labels in the same way that the regexp and now json pipelines allow.

Describe the solution you'd like Instead of extracting all labels from a logfmt-formatted line, I only want selected labels extracted.

The log line:

level=error ts=2021-02-12T19:18:10.037940878Z caller=client.go:294 component=client host=observability-loki-gateway msg="final error sending batch" status=400 error="server returned HTTP status 400 Bad Request (400): entry with timestamp 2021-02-12 19:18:08.917452 +0000 UTC ignored, reason: 'entry out of order' for stream..."

Will produce the following labels with a plain | logfmt pipeline:

• level=”error”
• ts=”2021-02-12T19:18:10.037940878Z”
• caller=”client.go:294”
• component=”client”
• host=”observability-loki-gateway”
• msg="final error sending batch"
• status=”400”
• error="server returned HTTP status 400 Bad Request (400): entry with timestamp 2021-02-12 19:18:08.917452 +0000 UTC ignored, reason: 'entry out of order' for stream..."

That’s a lot of labels!

If I only need the msg and status labels extracted, it would be nice to use the following:

{app=”foo”} | logfmt msg,status will produce:

• msg="final error sending batch"
• status=”400”

Or even {app=”foo”} | logfmt message="msg", status if I wanted to rename the label:

• message="final error sending batch"
• status=”400”

Describe alternatives you've considered N/A

Additional context

• Design doc

[hugree]

This is a very nice idea... same with JSON.

[btaani]

I would like to work on this one if no one else is

[dannykopping]

@btaani go for it :+1:

[danpoltawski]

👍 to this idea - when trying to do metric queries extracting all the logfmt labels creates a huge amount of cardinality

[winston0410]

New to LogQL. Does that mean that right now we have to output all value from logfmt? Can we filter out values there?

[btaani]

@winston0410 The goal of this feature is to be able to select only the desired labels from a log line instead of displaying all of them. The PR is already under review: #6675