Parser error

[satrushn]

Hello! I try to configure FluentBit-operator and I have an error:

[error] [input:tail:tail.0] parser 'docker' is not registered

What I don't understand in configuration?

My configuration is (in secret fluent-bit-config): fluent-bit.conf:

[Service] Http_Listen 0.0.0.0 Http_Port 2020 Http_Server true [Input] Name tail Path /var/log/containers/.log Refresh_Interval 10 Skip_Long_Lines true DB /fluent-bit/tail/pos.db DB.Sync Normal Mem_Buf_Limit 5MB Parser docker Tag kube. [Filter] Name kubernetes Match kube.* Kube_URL https://kubernetes.default.svc:443 Kube_CA_File /var/run/secrets/kubernetes.io/serviceaccount/ca.crt Kube_Token_File /var/run/secrets/kubernetes.io/serviceaccount/token Labels false Annotations false [Filter] Name nest Match kube.* Operation lift Nested_under kubernetes Add_prefix kubernetes_ [Filter] Name modify Match kube.* Remove stream Remove kubernetes_pod_id Remove kubernetes_host Remove kubernetes_container_hash Add productID TST.000 Add tenantID FBO Rename log message [Filter] Name nest Match kube.* Operation nest Wildcard kubernetes_* Nest_under kubernetes Remove_prefix kubernetes_ [Output] Name http Match * host logstash.sometest.svc.cluster.local port 5000 format json_lines json_date_key logDate json_date_format iso8601

parsers.conf:

[PARSER] Name docker Format json Time_Key time Time_Format %Y-%m-%dT%H:%M:%S.%L Time_Keep true

[wenchajun]

Docker parser is built into Fluentbit，so you don't need to create the Parser, you can delete it. https://github.com/fluent/fluentbit-operator/blob/master/conf/parsers.conf

[satrushn]

Thank you. Then after removing, how I must parse logs like this, field message: {"log":"2021-11-30 11:50:19.377 [INFO][39] int_dataplane.go 959: Applying dataplane updates\n","stream":"stdout","time":"2021-11-30T11:50:19.378077212Z"} ? Can you give an example please?

[wenchajun]

Thank you. Then after removing, how I must parse logs like this, field message: {"log":"2021-11-30 11:50:19.377 [INFO][39] int_dataplane.go 959: Applying dataplane updates\n","stream":"stdout","time":"2021-11-30T11:50:19.378077212Z"} ?

Maybe you can give the parser another name.like

Name dockermodle
Format json
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%L
Time_Keep true

But I'm not sure what your needs are.This field is in JSON format. You can handle log information through various Fluentbit plug-ins.

[satrushn]

I added a filter:

[PARSER] Name my-json Format json Time_Key time Time_Format %Y-%m-%dT%H:%M:%S.%L Time_Keep true

And now I see in logs the same:

message | {"log":"[2021/12/02 06:52:31] [ info] [storage] version=1.1.1, initializing...\n","stream":"stderr","time":"2021-12-02T06:52:31.435600219Z"}

I would like to see fields like:

log [2021/12/02 06:52:31] [ info] [storage] version=1.1.1, initializing...\n stream stderr time 2021-12-02T06:52:31.435600219Z

How must I configure in this case?

[wenchajun]

What environment do you use? kubesphere? or just Kubernetes? Maybe you can try

Name tail
Path /var/log/containers/.log
Refresh_Interval 10
Skip_Long_Lines true
DB /fluent-bit/tail/pos.db
DB.Sync Normal
Mem_Buf_Limit 5MB
Parser my-json
Tag kube.

[satrushn]

I use Kubernetes and input: [Input] Name tail Path /var/log/containers/.log Refresh_Interval 10 Skip_Long_Lines true DB /fluent-bit/tail/pos.db DB.Sync Normal Mem_Buf_Limit 5MB Parser docker Tag kube.

Information from tail of file is already in field 'message' message | {"log":"[2021/12/02 06:52:31] [ info] [storage] version=1.1.1, initializing...\n","stream":"stderr","time":"2021-12-02T06:52:31.435600219Z"} Then I would like to have this field divided to three different fields: 'log', 'stream', 'time'

'log' [2021/12/02 06:52:31] [ info] [storage] version=1.1.1, initializing...\n 'stream' stderr 'time' 2021-12-02T06:52:31.435600219Z

[wenchajun]

Maybe you can use this parser to modify it https://github.com/fluent/fluentbit-operator/blob/master/conf/parsers.conf#L125 In fact, this is fluentbit configuration, you can refer to fluentbit documentation to modify the configuration https://docs.fluentbit.io/manual/pipeline/inputs/tail