Hope the in_tail Input plugin support limit multiline log's size

[wangchaoforever]

Is your feature request related to a problem? Please describe.

When I collect logs which single line log's size is about 20M, it cause OOM KILL fluentd in in_tail stage. So I hope to have a parameter to limit single line log's size. Like collect single line log, When I collect multiline log, the OOM KILL is also happend in in_tail stage. So I hope to have a parameter to limit multiline log's size.

Describe the solution you'd like

I hope to have a parameter to limit the log's size in in_tail step. Just like the filter parameter which I can only keep the 50000 lenth from begin: <filter *> @type record_transformer enable_ruby message ${if record["message"].to_s.length > 50000 then record["message"][0..50000] else record["message"]; end}

Describe alternatives you've considered

Large single log or multiline logs will expend too much in source in_tail step, I don't want to skip the large log and hope to keep 50000 length. I have no idea to deal with it now.

Additional context

No response

[kenhys]

I hope that max_line_size will work for you. (It is available since Fluentd v1.14.4)

https://docs.fluentd.org/input/tail#max_line_size

[wangchaoforever]

@kenhys I upgrade my fluentd to v1.14.4 and use max_line_size in tail, but the OOM KILL still happend. The logs:

...
2022-03-22 15:36:03 +0800 [info]: starting fluentd-1.14.4 pid=6603 ruby="2.7.5"
2022-03-22 15:36:03 +0800 [info]: spawn command to main:  cmdline=["/opt/td-agent/bin/ruby", "-Eascii-8bit:ascii-8bit", "/opt/td-agent/bin/fluentd", "-c", "/etc/td-agent/td-agent-longlog.conf", "--under-supervisor"]
2022-03-22 15:36:03 +0800 [info]: #0 adding match pattern="test" type="stdout"
2022-03-22 15:36:04 +0800 [info]: #0 adding source type="tail"
2022-03-22 15:36:04 +0800 [info]: #0 starting fluentd worker pid=6608 ppid=6603 worker=0
2022-03-22 15:36:04 +0800 [info]: #0 following tail of /data/log/test/collect-log/test.log
2022-03-22 15:37:10 +0800 [warn]: #0 received line length is longer than 16384
2022-03-22 15:37:10 +0800 [warn]: #0 received line length is longer than 16384
2022-03-22 15:37:10 +0800 [info]: Worker 0 finished unexpectedly with signal SIGKILL
...

[github-actions[bot]]

This issue has been automatically marked as stale because it has been open 90 days with no activity. Remove stale label or comment or this issue will be closed in 30 days

[ashie]

@kenhys I upgrade my fluentd to v1.14.4 and use max_line_size in tail, but the OOM KILL still happend. The logs:

Is it multi line case? I believe max_line_size will fix the issue for single line case, and in fact you said it's effective in #3739. On the other hand, probably it won't skip big multi line log consist small single lines and might retain excessive memory.