ilogtail icon indicating copy to clipboard operation
ilogtail copied to clipboard

Refactor ProcessorSplitRegexNative to ProcessorMergeMultilineLogNative and Introduce ProcessorSplitNative for Line-based Event Splitting

Open quzard opened this issue 1 year ago • 0 comments

重构ProcessorSplitRegexNative插件为ProcessorMergeMultilineLogNative,原有的ProcessorSplitRegexNative是在同个content里进行多行切分。现在添加一个新的插件ProcessorSplitNative,用于将多行日志按照\n切分成多个event,然后再通过ProcessorMergeMultilineLogNative,按照多行正则配置,对event进行多行聚合。

此外ProcessorMergeMultilineLogNative插件增加一种行为——ProcessEventsWithPartLog。根据event里是否有PartLogFlag,来决定是否要合并event。例如如下:

{
    "events": [
        {
            "contents": {
                "PartLogFlag": "true",
                "content": "Exception"
            },
            "timestamp": 12345678901,
            "timestampNanosecond": 0,
            "type": 1
        },
        {
            "contents": {
                "PartLogFlag": "true",
                "content": " in thread"
            },
            "timestamp": 12345678902,
            "timestampNanosecond": 0,
            "type": 1
        },
        {
            "contents": {
                "PartLogFlag": "true",
                "content": "  'main'"
            },
            "timestamp": 12345678903,
            "timestampNanosecond": 0,
            "type": 1
        },
        {
            "contents": {
                "content": " java.lang.NullPoinntterException"
            },
            "timestamp": 12345678904,
            "timestampNanosecond": 0,
            "type": 1
        }
    ]
}

合并为:

{
    "events": [
        {
            "contents": {
                "content": "Exception in thread  'main' java.lang.NullPoinntterException"
            },
            "timestamp": 12345678901,
            "timestampNanosecond": 0,
            "type": 1
        }
    ]
}

quzard avatar Feb 05 '24 02:02 quzard