log with lengthy exception is getting dropped

[NareshAnaparthi]

Problem

Trying to concatenate Multi line records into single log record Working well for all kinds of exceptions/startup logs Lengthy exceptions are getting dropped. ...

Steps to replicate

Please tryout below config and seeing lengthy lines/exceptions are getting dropped. Would like to know if we can limit lines or increase if its related to Buffer limitation

Provide example config and message

2020-04-03T02:57:17.605Z ERROR

[Server] FATAL error: com.vmware.vdi.pop.configuration.ConfigurationException$IOConfigurationException: Failed to read configuration com.vmware.vdi.pop.configuration.ConfigurationException$IOConfigurationException: Failed to read configuration at com.vmware.vdi.pop.configuration.ConfigurationManager.read(ConfigurationManager.java:74) at com.vmware.vdi.pop.configuration.ConfigUtils.readConfig(ConfigUtils.java:88) at com.vmware.vdi.hydra.connector.server.Server.main(Server.java:287) Caused by: java.io.FileNotFoundException: ./tls/connector-api.config (No such file or directory) at java.io.FileInputStream.open0(Native Method) at java.io.FileInputStream.open(FileInputStream.java:195) at java.io.FileInputStream.(FileInputStream.java:138) at com.fasterxml.jackson.core.JsonFactory.createParser(JsonFactory.java:916) at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3070) at com.vmware.vdi.pop.configuration.ConfigurationManager.read(ConfigurationManager.java:68) ... 2 common frames omitted Native Memory Tracking: Total: reserved=2315290KB, committed=68470KB

•             Java Heap (reserved=983040KB, committed=32768KB)
                        (mmap: reserved=983040KB, committed=32768KB)
  

•                 Class (reserved=1063273KB, committed=16489KB)
                        (classes #2827)
                        (malloc=361KB #1208)
                        (mmap: reserved=1062912KB, committed=16128KB)
  

•                Thread (reserved=5718KB, committed=5718KB)
                        (thread #10)
                        (stack: reserved=5672KB, committed=5672KB)
                        (malloc=33KB #52)
                        (arena=13KB #19)
  

•                  Code (reserved=250014KB, committed=3342KB)
                        (malloc=414KB #1119)
                        (mmap: reserved=249600KB, committed=2928KB)
  

•                    GC (reserved=3215KB, committed=123KB)
                        (malloc=7KB #84)
                        (mmap: reserved=3208KB, committed=116KB)
  

•              Compiler (reserved=138KB, committed=138KB)
                        (malloc=7KB #90)
                        (arena=131KB #5)
  

•              Internal (reserved=506KB, committed=506KB)
                        (malloc=474KB #3772)
                        (mmap: reserved=32KB, committed=32KB)
  

•                Symbol (reserved=4910KB, committed=4910KB)
                        (malloc=3815KB #30989)
                        (arena=1095KB #1)
  

• Native Memory Tracking (reserved=589KB, committed=589KB) (malloc=3KB #35) (tracking overhead=586KB)

•           Arena Chunk (reserved=3887KB, committed=3887KB)
                        (malloc=3887KB)_
  

Expected Behavior

concat plugin should be able concat above multiline exception/message into single log record. OR Need a way to truncate instead of dropping entire message. ...

Your environment

Concat Plugin Config <filter **> @type concat key log multiline_start_regexp /^\d{4}[-/]\d{1,2}[-/]\d{1,2}/ separator "" flush_interval 5

• OS version Linux dev1-westus2-jumpbox 5.0.0-1028-azure #30~18.04.1-Ubuntu SMP Fri Dec 6 11:47:59 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

• paste result of fluentd --version or td-agent --version

• plugin version

  • paste boot log of fluentd or td-agent
  • paste result of fluent-gem list, td-agent-gem list or your Gemfile.lock fluent/fluentd-kubernetes-daemonset:v1-debian-elasticsearch

[kaisermario]

Any updates on this? Having also issues with concatenation of multineline logs and multiline_start_regexp.

[okkez]

Hmm, I cannot reproduce the issue.