Support structured key-value pairs in JSON layout

[martin-tarjanyi]

For a while Logback has had dedicated support to log and print key-value pairs (%kvp in PatternLayout) to facilitate structured logging. It looks like currently the Loki appender doesn't print these pairs when using the built-in JSON layout.

Thanks to the extensibility I was able to implement a custom JSON provider (based on the MdcJsonProvider) which does the job. Due to my project it's written in Kotlin:

class KeyValuePairJsonProvider : AbstractFieldJsonProvider() {
    init {
        fieldName = "kvp_"
    }

    override fun canWrite(event: ILoggingEvent): Boolean {
        return !event.keyValuePairs.isNullOrEmpty()
    }

    override fun writeTo(
        writer: JsonEventWriter,
        event: ILoggingEvent,
        startWithSeparator: Boolean,
    ): Boolean {
        val keyValuePairs = event.keyValuePairs
        var firstFieldWritten = false
        for (keyValue in keyValuePairs) {
            val key = keyValue.key
            val value = keyValue.value
            // skip empty records
            if (key == null || value == null) continue

            if (startWithSeparator || firstFieldWritten) writer.writeFieldSeparator()
            writer.writeStringField(fieldName + key, value.toString())
            firstFieldWritten = true
        }
        return firstFieldWritten
    }

    override fun writeExactlyOneField(
        writer: JsonEventWriter,
        event: ILoggingEvent,
    ) {
        throw UnsupportedOperationException(
            "KeyValuePairJsonProvider can write an arbitrary number of fields. " +
                "`writeExactlyOneField` should never be called for KeyValuePairJsonProvider.",
        )
    }
}

One important caveat is that unlike MDC values which can only be String, values in these pairs could be any Object. My implementation for now just calls a toString on the value so nested objects are not rendered as JSON.

For reference there is a logstash encoder which implemented this with nested object serialization support: https://github.com/logfellow/logstash-logback-encoder/blob/b1e7653914fba93baf6042b15d44e54909879e4f/src/main/java/net/logstash/logback/composite/loggingevent/KeyValuePairsJsonProvider.java

[nehaev]

Hi @martin-tarjanyi, thanks for reporting this! It's great to hear that you've found this new functionality useful!

KeyValuePair is indeed tricky to implement. Logstash encoder seems to solve nested object serialization problem using Jackson (e.g., reflection). I think that neither bundling with Jackson, nor implementing a reflection-based JSON serializer from scratch is a good idea for Loki4j. Probably we can come up with an implementation that supports standard types (int, boolean, etc.) and does toString() for the rest, but I'm not sure it will be much better than no implementation at all. No implementation means you can write a custom one, adjusted to your particular use case (i.e., exactly what you did).