Conversion Word for logfmt based markers

[wsargent]

Markers are great when outputting JSON, but when rendering in a line based format, they don't do so well.

As an example, here's the output in JSON:

{"id":"FgEdic8a562dHaINzdiAAA","relative_ns":55243400,"tse_ms":1585963041084,"start_ms":null,"@timestamp":"2020-04-04T01:17:21.084Z","@version":"1","message":"herp derp  arg1=value1 numericArg=42 booleanArg=false a=b sequenceArg=[a, b, c]","logger_name":"example.Main$","thread_name":"main","level":"INFO","level_value":20000,"string":"steve","number":42,"boolean":true,"markerJson":{
  "f1" : "v1"
},"array":["one","two","three"],"arg1":"value1","numericArg":42,"booleanArg":false,"a":"b","sequenceArg":["a","b","c"]}

and here's the output in line based format, using %marker:

FgEdic8a562dHaINzdiAAA 2020-04-04T01:17:21.084+0000 [INFO ] example.Main$ in main HashSet(string=steve, number=42, boolean=true, DEFERRED, array=[one, two, three]) [ LS_APPEND_OBJECT, LS_APPEND_OBJECT, LS_APPEND_OBJECT, DEFERRED, LS_APPEND_OBJECT ] - herp derp  arg1=value1 numericArg=42 booleanArg=false a=b sequenceArg=[a, b, c]

It'd be great to have a conversion word that turned the markers into logfmt:

• https://brandur.org/logfmt
• https://rollout.io/blog/logfmt-a-log-format-thats-easy-to-read-and-write/

[philsttr]

The markers provided by logstash-logback-encoder support complex JSON output: nested objects, arrays, raw json, etc. Whereas logfmt only supports key/value pairs. How do you propose handling complex types in logfmt?

[wsargent]

That's a good question. I think what you're doing right now with StructuredArguments is actually the correct answer -- if logfmt only supports key/value pairs, then any line oriented representation of more complex data is going to be de facto an extension.

The catch to this is that I don't think the format of complex StructuredArguments is explicitly documented, and that probably needs to be a thing.

[philsttr]

The catch to this is that I don't think the format of complex StructuredArguments is explicitly documented, and that probably needs to be a thing.

Hmm, yeah, it's not totally spelled out in the README.md, but it's in the javadoc. The readme could definitely use some work to make it more clear.