fluent-bit icon indicating copy to clipboard operation
fluent-bit copied to clipboard

Missing record keys when using multiline java parser

Open pfrcks opened this issue 1 year ago • 0 comments

Bug Report

Describe the bug Some output records do not contain keys like "filepath", "_p", "time", stream. Eg record [2] below

[0] tail.0: [[1708736407.169656515, {}], {"time"=>"2024-02-24T01:00:07.169656515Z", "stream"=>"stdout", "_p"=>"F", "log"=>"2022/11/29 05:41:10 ConsoleApplication1.MyCustomException: some message .... ---> System.Exception: Oh noes!
   at ConsoleApplication1.SomeObject.OtherMethod() in C:\ConsoleApplication1\SomeObject.cs:line 24
   at ConsoleApplication1.SomeObject..ctor() in C:\ConsoleApplication1\SomeObject.cs:line 14
   --- End of inner exception stack trace ---
   at ConsoleApplication1.SomeObject..ctor() in C:\ConsoleApplication1\SomeObject.cs:line 18
   at ConsoleApplication1.Program.Main(String[] args) in C:\ConsoleApplication1\Program.cs:line 13", "filepath"=>"/var/log/containers/dotnet-qk7vm_default_dotnet-c8de97360d3de86d36bddfedf77392cd817b7c587d0bb16b1ece4c388ea7cef0.log"}]
[1] tail.0: [[1708736408.170722067, {}], {"time"=>"2024-02-24T01:00:08.170722067Z", "stream"=>"stdout", "_p"=>"F", "log"=>"2022/11/29 05:41:10 StackTrace: '   at System.Environment.GetStackTrace(Exception e)", "filepath"=>"/var/log/containers/dotnet-qk7vm_default_dotnet-c8de97360d3de86d36bddfedf77392cd817b7c587d0bb16b1ece4c388ea7cef0.log"}]
[2] tail.0: [[1708736407.169656515, {}], {"log"=>"   at System.Environment.GetStackTrace(Exception e)
   at System.Environment.get_StackTrace()
   at Sample.Main()'"}]
[3] tail.0: [[1708736408.171001925, {}], {"time"=>"2024-02-24T01:00:08.171001925Z", "stream"=>"stdout", "_p"=>"F", "log"=>"2022/11/29 05:41:10 normal log", "filepath"=>"/var/log/containers/dotnet-qk7vm_default_dotnet-c8de97360d3de86d36bddfedf77392cd817b7c587d0bb16b1ece4c388ea7cef0.log"}]

To Reproduce

  • Rubular link if applicable:
  • Example log message if applicable:
  • Steps to reproduce the problem:
2022/11/29 05:41:10 ConsoleApplication1.MyCustomException: some message .... ---> System.Exception: Oh noes!
   at ConsoleApplication1.SomeObject.OtherMethod() in C:\ConsoleApplication1\SomeObject.cs:line 24
   at ConsoleApplication1.SomeObject..ctor() in C:\ConsoleApplication1\SomeObject.cs:line 14
   --- End of inner exception stack trace ---
   at ConsoleApplication1.SomeObject..ctor() in C:\ConsoleApplication1\SomeObject.cs:line 18
   at ConsoleApplication1.Program.Main(String[] args) in C:\ConsoleApplication1\Program.cs:line 13
2022/11/29 05:41:10 StackTrace: '   at System.Environment.GetStackTrace(Exception e)
   at System.Environment.GetStackTrace(Exception e)
   at System.Environment.get_StackTrace()
   at Sample.Main()'
2022/11/29 05:41:10 normal log

save the logs above to a file and use as input for tail plugin with the config below.

Expected behavior Output record should have the necessary keys

Screenshots

Your Environment

  • Version used: tested on linux 2.0.9 and 2.1.10
  • Configuration:
[INPUT]
    Name tail
    Path /var/log/containers/*.log
    Read_from_Head true
    Multiline.Parser cri
    Mem_Buf_Limit 10m
    Buffer_Chunk_Size 1m
    Buffer_Max_Size 1m
    Rotate_Wait 20
    Refresh_Interval 30
    Path_Key filepath
    Skip_Long_Lines On


[FILTER]
    Name multiline
    Match *
    multiline.key_content log
    multiline.parser java
  • Environment name and version (e.g. Kubernetes? What version?):
  • Server type and version:
  • Operating System and version:
  • Filters and plugins:

Additional context

pfrcks avatar Feb 24 '24 04:02 pfrcks