Multiline logs are not shown as one Splunk log event but each line is shown as a Splunk log event

[minhpham-westpac]

What happened: In the values.yaml for logging, the pod and container are specified as following with test-sys as the namespace, we call it part A: aks-secrets-store-provider-azure: from: pod: test-sys/aks-secrets-store-provider-azure- container: provider-azure-installer multiline: firstline: /^\w[0-1]\d[0-3]\d/ endline: / pod=".+"/ separator: "\n" flushInterval: 5 sourcetype: kube:secrets-store-provider-azure

The pod aks-secrets-store-provider-azure generates logs as following, we call it part B:

C1234 02:06:52.093686 1 provider.go:196] "objects string defined in secret provider class" objects=<
array: - |
objectName: xyz-key objectType: secret # object types: secret, key or cert
objectVersion: "" # [OPTIONAL] object versions, default to latest if empty

• |
  objectName: xyz-id
  objectType: secret # object types: secret, key or cert objectVersion: "" # [OPTIONAL] object versions, default to latest if empty

pod="kube-system/aks-cluster-compliance-rcg87"

What you expected to happen: The above multiline log in part B should be displayed in one Splunk log event as the following, we call it part C

C1234 02:06:52.093686 1 provider.go:196] "objects string defined in secret provider class" objects=<
array: - |
objectName: xyz-key objectType: secret # object types: secret, key or cert
objectVersion: "" # [OPTIONAL] object versions, default to latest if empty

• |
  objectName: xyz-id
  objectType: secret # object types: secret, key or cert objectVersion: "" # [OPTIONAL] object versions, default to latest if empty

pod="kube-system/aks-cluster-compliance-rcg87"

But instead Splunk shows each line as a separate log event as the following, we call it part C:

23/05/2023 17:00:00.123 C1234 02:06:52.093686 1 provider.go:196] "objects string defined in secret provider class" objects=<
23/05/2023 17:00:00.124 array: 23/05/2023 17:00:00.125 - | 23/05/2023 17:00:00.126 objectName: xyz-key .... 23/05/2023 17:00:00.200 > pod="kube-system/aks-cluster-compliance-rcg87"

How to reproduce it (as minimally and precisely as possible): Using the above config for the AKS container in part A, and the input is the multiline log in part B,

Anything else we need to know?: I follow exactly the instructions in the Splunk connect for Kubernetes https://github.com/splunk/splunk-connect-for-kubernetes/issues?q=is%3Aissue+is%3Aclosed+multiline, and the multiline log is not shown as one Splunk log event. I do not know why, please help me to have the function working. Thanks very much for your help in advance.

Environment:

• Kubernetes version (use kubectl version): Kubernetes v1.26.3
• Ruby version (use ruby --version): Not use Ruby at all
• OS (e.g: cat /etc/os-release): Red Hat Enterprise Linux Server, VERSION="7.9 (Maipo)"
• Splunk version:
• Splunk Connect for Kubernetes helm chart version: Splunk connect for k8s 1.5.3
• Others:

Please get back to me if you have further questions, or clarification. I am looking forward to your solutions and Thanks very much for your help in advance.