Adding Layout to logback-ecs-encoder

[dgempiuc]

trafficstars

If there is an error in incoming requests, I am logging this request and some requests may contain sensitive information, such as password.

There is LayoutWrappingEncoder encoder on logback and I am able to add Layout that.

        <encoder class="ch.qos.logback.core.encoder.LayoutWrappingEncoder">
            <layout class="org.custom.ecs.layout.MaskingPatternLayout">
                <maskPattern>\"oldPassword\"\s*:\s*\"(.*?)\"</maskPattern> <!-- password pattern -->
                <maskPattern>\"newPassword\"\s*:\s*\"(.*?)\"</maskPattern> <!-- password pattern -->
                <pattern>${CONSOLE_LOG_PATTERN}</pattern>
            </layout>
        </encoder>

payload={ "newPassword": "********", "oldPassword": "********", "smsCode": "123456" }

But, I am using EcsEncoder in production environment, but there is no option to add Layout in this encoder. So, I can't mask fields. Is there anyway to do that or is there any plan to add this?

[dgempiuc]

There is encode method on EcsEncoder and this method creates the log message to be written to the file.

    @Override
    public byte[] encode(ILoggingEvent event) {
        ...
        EcsJsonSerializer.serializeFormattedMessage(builder, event.getFormattedMessage());
        ...
     }

If we add Layout field in EcsEncoder class, we can use this in encode method.

public class EcsEncoder {

    ...
    protected Layout<ILoggingEvent> layout;

    @Override
    public byte[] encode(ILoggingEvent event) {
        StringBuilder builder = new StringBuilder();
        ...
        EcsJsonSerializer.serializeFormattedMessage(builder, this.layout.doLayout(event));
        ...
     }

     ...

    public Layout<ILoggingEvent> getLayout() {
        return this.layout;
    }

    public void setLayout(Layout<ILoggingEvent> layout) {
        this.layout = layout;
    }

}

So, we can use our custom Layout or predefined Layout.

        <encoder class="co.elastic.logging.logback.EcsEncoder">
            <serviceName>${springAppName:-}</serviceName>
            <layout class="org.custom.ecs.layout.MaskingEcsPatternLayout">
                <maskPattern>\"oldPassword\"\s*:\s*\"(.*?)\"</maskPattern> <!-- password pattern -->
                <maskPattern>\"newPassword\"\s*:\s*\"(.*?)\"</maskPattern> <!-- password pattern -->
                <pattern>${CONSOLE_LOG_PATTERN}</pattern>
            </layout>
        </encoder>

Here is my custom Layout.

public class MaskingEcsPatternLayout extends PatternLayout {

    private Pattern multilinePattern;
    private List<String> maskPatterns = new ArrayList<>();

    public void addMaskPattern(String maskPattern) {
        maskPatterns.add(maskPattern);
        multilinePattern = Pattern.compile(maskPatterns.stream().collect(Collectors.joining("|")), Pattern.MULTILINE);
    }

    @Override
    public String doLayout(ILoggingEvent event) {
        return maskMessage(event.getFormattedMessage());
    }

    public String maskMessage(String message) {
        if (multilinePattern == null) {
            return message;
        }
        StringBuilder sb = new StringBuilder(message);
        Matcher matcher = multilinePattern.matcher(sb);
        while (matcher.find()) {
            IntStream.rangeClosed(1, matcher.groupCount()).forEach(group -> {
                if (matcher.group(group) != null) {
                    IntStream.range(matcher.start(group), matcher.end(group)).forEach(i -> sb.setCharAt(i, '*'));
                }
            });
        }
        return sb.toString();
    }

}

[dgempiuc]

created pr https://github.com/elastic/ecs-logging-java/pull/145

[Tokimimo]

Any chance such a function will be added? Or am I maybe missing something to achieve the above with the current encoder or available resources?

[michalgutkowski]

@Tokimimo IMO it works as expected after recent changes in #145:

 <appender name="ECS_JSON_CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
    <encoder class="co.elastic.logging.logback.EcsEncoder">
        <messageLayout class="ch.qos.logback.classic.PatternLayout">
            <pattern>[%d{HH:mm:ss.SSS}] %t %-5p %c  - %m%n %rEx{full, org.springframework}</pattern>
        </messageLayout>
    </encoder>
</appender>

[rdifrango]

This was closed by #220 which was built upon #145 and has since been merged. @felixbarny and @mgreau we should close this issue now that it's been resolved.