in_tail plugin can cause breakdowns in fluentd

[yugeeklab]

Describe the bug

A requirement for our work is to read a 1GB single-line text. This text is generated by PostgreSQL, which has no limit for the log. However, Fluentd breaks down when attempting to read this 1GB single-line text.

I believe this could be considered a bug and vulnerability.

To Reproduce

Add below code in test_in_tail.rb

  def test_max_line_size_with_large_singleline
    label = "1k"
    size = 1000000000
    config = config_element("", "", {
                          "tag" => "max_line_size",
                          "path" => "#{@tmp_dir}/with_long_lines.txt",
                          "format" => "none",
                          "read_from_head" => true,
                          "max_line_size" => label,
                          "log_level" => "debug"
                        })
    Fluent::FileWrapper.open("#{@tmp_dir}/with_long_lines.txt", "w+") do |f|
      f.puts "foo"
      f.puts "x" * size # 'x' * size + \n > @max_line_size
      f.puts "bar"
    end
    d = create_driver(config, false)
    timestamp = Time.parse("Mon Nov 29 11:22:33 UTC 2021")
    Timecop.freeze(timestamp)
    d.run(expect_records: 2)
    assert_equal([
                    [{"message" => "foo"},{"message" => "bar"}],
                    [
                      "2021-11-29 11:22:33 +0000 [warn]: received line length is longer than #{size}\n",
                      "2021-11-29 11:22:33 +0000 [debug]: skipped line\n"
                    ]
                  ],
                  [
                    d.events.collect { |event| event.last },
                    d.logs[-2..]
                  ])
  end

Run below command

bundle exec rake test TEST=test/plugin/test_in_tail.rb TESTOPTS="--name=test_max_line_size_with_large_singleline"

Add below code in test_in_tail.rb

  def test_max_line_size_with_large_multiline
    Fluent::FileWrapper.open("#{@tmp_dir}/tail.txt", "wb") { |f| }

    config = config_element("", "", {
                              "format" => "multiline",
                              "format_firstline" => "/^s/",
                              "format1" => "/^s (?<message1>[^\\n]+)\\n?/",
                              "format2" => "/^f (?<message2>[^\\n]+)\\n?/",
                              "max_line_size" => "1k",
                              "multiline_flush_interval" => "1s",
                          })
    d = create_driver(config)
    msg = "t" * 1000000000
    d.run(expect_emits: 1) do
      Fluent::FileWrapper.open("#{@tmp_dir}/tail.txt", "ab") { |f|
        f.puts "s #{msg}"
        f.puts "f #{msg}"
        f.puts "s test1"
        f.puts "f test2"
      }
    end

    events = d.events

    assert_equal({"message1" => "test1", "message2" => "test2"}, events[0][2])
    assert_equal(1, events.length)
  end

Run below command

bundle exec rake test TEST=test/plugin/test_in_tail.rb TESTOPTS="--name=test_max_line_size_with_large_multiline"

Expected behavior

test_max_line_size_with_large_singleline

AS-IS : it passed. but it took much long time. Finished in 2365.724927 seconds.

TO-BE : it works. Finished in 0.915693 seconds.

test_max_line_size_with_large_multiline

AS-IS : it failed. Finished in 305.73504 seconds.

TO-BE : it works. Finished in 24.163525 seconds

Your Environment

- Fluentd version: master branch
- TD Agent version: 
- Operating system: Monterey 12.6.1
- Kernel version: 21.6.0

Your Configuration

None

Your Error Log

test_max_line_size_with_large_multiline

Error: test_max_line_size_with_large_multiline(TailInputTest): NoMethodError: undefined method `[]' for nil

Additional context

No response