JSON logging as alternative to pattern layout logging

[mp911de]

Spring Boot offers logging using a pattern for various log frameworks. In scenarios, where more context details would be useful, users need to adopt the pattern. Each value from the MDC needs to be configured independently which is a consistent source of work. It would be great to opt-in for a JSON log format that is used instead of the pattern layout to have a flexible field schema for later processing within log aggregators such as Splunk, Graylog, or Logstash.

Use-cases

• Using MDC to store contextual details helps while tracing and debugging applications. Currently, the pattern layout is configured in a static way (in config files) while MDC values can be added dynamically. Adding dynamically values to the JSON structure from the MDC does not necessarily require additional configuration each time a new MDC entry is added to the application. That's actually the use case from which the power comes.
• Log forwarders need to be adopted for each change of the log format. Adding Spring Clouds Sleuth to use the slf4j span logger requires for instance adoption of the log format.
• Multiline logging: Configuring log forwarders for parsing multiline logs is challenging. Exceptions can be nicely wrapped within a JSON field; The syntax of JSON makes it easy because no additional multiline configuration needs to be set up
• JSON logs can be either picked up from stdout by an online-forwarder or written into a file that is read by logstash forwarder (or other tools)

See https://github.com/mp911de/logstash-gelf for examples.

[cemo]

This would be a nice addition. I had to customize https://github.com/logstash/logstash-logback-encoder project to integrate with Logentries which is also supporting JSON logs.

Please note that logstash-logback-encoder is general-purpose JSON logging mechanism project.

[philwebb]

@mp911de Is there anything that's currently preventing JSON logging from working for you? I'm not sure that we really want to take on too much more responsibility in this area. We already have quite a maintenance burden trying to support three different logging systems.

[mp911de]

There's nothing which prevents JSON logging/log forwarding. It's more about the barrier to configure logging and set up conditional JSON logging. Usually, in dev, one does not want to have JSON logs but would enable JSON logging in staging/production. Using different logging config files and specifying logging.config=(my config file) per environment increases the complexity.

I fully understand the maintenance efforts from your side. I'm considering maybe another approach with building/providing a starter in my own repo as I did JSON logging earlier. This way an extension existed one could use, and you don't have the maintenance hassle.

[philwebb]

That would certainly be better from our point of view. We could always add other extension points if needed.

[mp911de]

I'll follow that path then, thanks for your feedback.

[imjorge]

@mp911de have you gotten to implement the spring boot starter you mentioned?

[mp911de]

@imjorge I haven't been doing much on that end.

[pinfrederic]

Hi @cemo When you say that you include the log stash plugin : https://github.com/logstash/logstash-logback-encoder How did you plug Logentries with the LoggingEventCompositeJsonEncoder from this lib ?

Here is an extract of my logback conf:

    <appender name="loggly" class="com.rapid7.logback.LogentriesAppender">
        <Token>MY_TOKEN</Token>
        <Region>eu</Region>
        <Debug>False</Debug>
        <Ssl>True</Ssl>
        <facility>USER</facility>
        <!--<layout>-->
            <!--<pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n</pattern>-->
        <!--</layout>-->

        <encoder class="net.logstash.logback.encoder.LoggingEventCompositeJsonEncoder">
            <providers>
                <!-- Log all StructuredArgument instances -->
                <arguments/>
                <!-- Log all MDC fields except the ones from Sleuth - we add them below
                under cleaned up names -->
                <mdc>
                    <excludeMdcKeyName>X-B3-TraceId</excludeMdcKeyName>
                    <excludeMdcKeyName>X-B3-SpanId</excludeMdcKeyName>
                    <excludeMdcKeyName>X-B3-ParentSpanId</excludeMdcKeyName>
                    <excludeMdcKeyName>X-Span-Export</excludeMdcKeyName>
                </mdc>
                <!-- Include Exception stack traces -->
                <stackTrace/>
                <timestamp>
                    <timeZone>UTC</timeZone>
                </timestamp>
                <!-- Assign logger fields to JSON object -->
                <pattern>
                    <pattern>
                        {
                        "component": "BackendJava",
                        "endpoint": "%mdc{method} | %mdc{url}",
                        "severity": "%level",
                        "params": "%mdc{params}",
                        "mcdTest": "%mdc{mdcData}",
                        "thread": "%thread",
                        "logger": "%logger",
                        "message": "%message"
                        }
                    </pattern>
                </pattern>
            </providers>
        </encoder>

    </appender>

But I always get this error: A "net.logstash.logback.encoder.LoggingEventCompositeJsonEncoder" object is not assignable to a "ch.qos.logback.classic.encoder.PatternLayoutEncoder" variable.

Thanks

[joshiste]

Imho it is quite a common practice to have logs in json format, so it can be easily digested by logstash, graylog, splunk, etc.

I think would be nice If spring boot would provide a simple option to switch the log format to json (and disable the banner at the same time). Currently you need to add some dependencies and override the entire log configuration. It would be nice to have a starter and a switch for it. The switch would be useful to change the setting for local development vs. deployment.

Logback

You currently need to add the net.logstash.logback:logstash-logback-encoder dependency, and override the entire logback configuration. If you include org/springframework/boot/logging/logback/base.xml you have the problem that you can't remove the appenders already included there. Also in contrast to the DefaultLogbackConfiguration a file appender is instantiated writing to some temp file (which may be undesirable in container environments). The DefaultLogbackConfiguration itself is not customizable and if you want to make the logback.xml configuration conditional on some property you need to also include the org.codehaus.janino:janino dependency.

Log4j2

you need to add jackson as dependency. And replace the entire configuration using an appender using the JsonLayout loosing all the defaults from spring boot if not duplicated. For having a switch you can add Routing to the configuration.

[rgoers]

Log4j 2.14.0 will include a new component, JsonTemplateLayout that uses a template to describe the format of the JSON. It was donated by the author of https://github.com/vy/log4j2-logstash-layout who is now a committer on the Log4j project. As I recall the implementation does not require Jackson, although I would be surprised if Jackson is an issue for anyone using Spring Boot.

[csterwa]

We have had requests for integration with log tooling in Azure for JSON pattern layout. Is this going to be making it into an upcoming release?

[jamietanna]

@csterwa ahead of that it's possible to do it ie via https://www.jvt.me/posts/2021/05/31/spring-boot-structured-logging/

[philwebb]

@csterwa No imminent plans I'm afraid. Boot 3 / Framework 6 work will be taking priority.

[davidkarlsen]

@philwebb Activating a profile would bring this a long way in order to be able to do something like:

<?xml version="1.0" encoding="UTF-8"?>
<configuration>

  <include resource="org/springframework/boot/logging/logback/defaults.xml" />
  <contextListener class="ch.qos.logback.classic.jul.LevelChangePropagator" />

  <springProfile name="kubernetes">
    <appender name="jsonAppender" class="ch.qos.logback.core.ConsoleAppender">
      <encoder class="net.logstash.logback.encoder.LogstashEncoder"/>
    </appender>
  </springProfile>

  <springProfile name="!kubernetes">
    <include resource="org/springframework/boot/logging/logback/console-appender.xml" />
  </springProfile>

  <root level="INFO">
    <appender-ref ref="CONSOLE" />
  </root>

</configuration>

I am aware that spring-cloud-kubernetes can define the profile, but I want to avoid pulling all of that in. I am also aware of https://docs.spring.io/spring-boot/docs/current/api/org/springframework/boot/cloud/CloudPlatform.html - but AFAIK that can't be used for toggling in spring-logback.xml.

WDYT?

[wilkinsona]

See https://github.com/spring-projects/spring-boot/issues/39814 for an example of a subtle bug that can occur with certain json logging configurations.

[wilkinsona]

Re-opening to address the test failures on Windows and to remind us to add something to the release notes.