loggie icon indicating copy to clipboard operation
loggie copied to clipboard

add grok processor in normalize interceptor

Open ethfoo opened this issue 2 years ago • 2 comments

Describe the feature

Loggie has many built-in processors such as regex/split in normalize interceptor for structured logs formats, and we hope to add a grok processor as an easy way for parsing logs.

针对日志切分的场景,Loggie现在在normalize interceptor里有regex、split等processor,我们希望可以支持grok processor。

ethfoo avatar May 16 '22 08:05 ethfoo

我来

evanzhang87 avatar May 26 '22 12:05 evanzhang87

配置格式

type GrokConfig struct {
	Target            string            `yaml:"target,omitempty" default:"body"` 
	Dst               string            `yaml:"dst,omitempty"`
	Match             []string          `yaml:"match,omitempty" validate:"required"`
	IgnoreBlank       bool              `yaml:"ignore_blank,omitempty" default:"true"`
	PatternPaths      []string          `yaml:"pattern_paths,omitempty"`
	Overwrite         bool              `yaml:"overwrite,omitempty" default:"true"`
	Pattern           map[string]string `yaml:"pattern,omitempty"`
	UseDefaultPattern bool              `yaml:"use_default_pattern,omitempty" default:"true"`
}
Target : 目标字段,默认body
Dst :  结果字段,默认为空,直接写到head里,填了值dst之后会以 dst为key,整个解析结果为value,写到head
Match :grok表达式,不能为空
IgnoreBlank :  是否忽略空字段,默认为true,如果解析得到的字段key的结果为“”,那么结果不会写入key:""
PatternPaths :获取pattern的路径,支持url和path,其中url为解析get请求的response,这里提供一个实例的url: https://raw.githubusercontent.com/vjeantet/grok/master/patterns/grok-patterns;path则为本地路径,如果填写的是目录则会拿目录下所有文件内可能包含的规则
Overwrite : 是否覆写,默认true,优先级关系为: 自定义pattern > PatternPaths(跟配置的list顺序有关,越靠后越优先) > DefaultPattern,优先级高的会覆盖低的
Pattern : 自定义pattern  
UseDefaultPattern : 是否使用预定义的Pattern,默认为true。(这个选项是不是可以删除,可以让选项一直为true)

evanzhang87 avatar May 28 '22 12:05 evanzhang87