No line breaks in multiline Java output after switching to EKS 1.21 and containerd.

[ftpd]

Last week I've updated my EKS cluster to 1.21 and enabled containerd. Logging works (I had to set pathDest to /var/log/pods and logFormatType to cri), but I have one issue: my multiline Java stacks previously had line breaks inside, which caused the output was easy readable for humans. Now it's all in one line and it's hard to read. Is there something I could do in splunk-connect itself or it's just 'a feature, not a bug' in CRI and there is nothing I can do?

Here is my current regexp for multiline:

firstline: /^\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}|^\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}|^\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3}\s-\s-/

and I also tried with it disabled, but still to not avail.

The bad sign (I suppose?) is I have no line breaks in raw log stream on the node itself. Is there anything I could adjust?

[ftpd]

Maybe to be more precise, here is some output:

The 'before' state, where line breaks were respected:

2022-07-06 11:07:07.336 ERROR 12 --- [3 - domainevent] c.c.t.messaging.sqs.CompanyRouteBuilder      : [] Failed delivery for [....]

Message History (source location and message history is disabled)
---------------------------------------------------------------------------------------------------------------------------------------
Source                                   ID                             Processor                                          Elapsed (ms)
[...]

The 'after' state, where line breaks are totally omitted:

2022-07-26 09:33:34.284 ERROR 11 --- [7 - domainevent] c.c.t.messaging.sqs.CompanyRouteBuilder      : [] Failed delivery for [...]Message History (source location and message history is disabled)---------------------------------------------------------------------------------------------------------------------------------------Source                                   ID                             Processor                                          Elapsed (ms)[...]

Raw log stream on the node running pod

2022-07-25T13:22:48.285422882Z stdout F 2022-07-25 13:22:48.284 ERROR 11 --- [rRedeliveryTask] c.c.t.messaging.sqs.CompanyRouteBuilder      : [] Failed delivery for [...]
2022-07-25T13:22:48.285457185Z stdout F
2022-07-25T13:22:48.285463662Z stdout F Message History (source location and message history is disabled)
2022-07-25T13:22:48.28546952Z stdout F ---------------------------------------------------------------------------------------------------------------------------------------
2022-07-25T13:22:48.285479731Z stdout F Source                                   ID                             Processor                                          Elapsed (ms)
2022-07-25T13:22:48.285492265Z stdout F [...]

(Of course there is a lot more in places I've put [...], but as it's sensitive/internal data, I had to make them redacted a bit.

The only change after switching to EKS 1.21 with containerd was adding the following block:

containers:
  pathDest: /var/log/pods
  logFormatType: cri

to my values.yaml file (as previously we had json from pure docker). I've also tested with /var/log/containers, but there is no difference anyway.

[hvaghani221]

Hi @ftpd, sorry for the delayed response. Actually, it is expected behaviour. You can use provide a line separator as well. It is not documented, so need to add one.

Sample config:

logs:
    <name>:
      from: 
        pod: <pod-name>
      multiline:
        firstline: /<regex>/
        separator: "\n"