fluentd icon indicating copy to clipboard operation
fluentd copied to clipboard
verified publisher icon fluent

Quoted string parser reports end of file when encountering an empty line

Open Alainx277 opened this issue 1 year ago • 4 comments

Describe the bug

When quoting a string it is possible to have text that goes on multiple lines. If the multiline string contains an empty line however, the parser fails even if the string is properly closed.

To Reproduce

Create a fluentd config like this:

<filter test>
  @type record_transformer
  <record>
    hello "world

"
  </record>
</filter>

Run fluentd in dry-run mode:

fluentd --dry-run -c /tmp/fluent.conf
Traceback (most recent call last):
	18: from /Users/me/.rbenv/versions/2.5.1/bin/fluentd:23:in `<main>'
	17: from /Users/me/.rbenv/versions/2.5.1/bin/fluentd:23:in `load'
	16: from /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/fluentd-1.16.6/bin/fluentd:15:in `<top (required)>'
	15: from /Users/me/.rbenv/versions/2.5.1/lib/ruby/site_ruby/2.5.0/rubygems/core_ext/kernel_require.rb:85:in `require'
	14: from /Users/me/.rbenv/versions/2.5.1/lib/ruby/site_ruby/2.5.0/rubygems/core_ext/kernel_require.rb:85:in `require'
	13: from /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/fluentd-1.16.6/lib/fluent/command/fluentd.rb:351:in `<top (required)>'
	12: from /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/fluentd-1.16.6/lib/fluent/supervisor.rb:624:in `configure'
	11: from /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/fluentd-1.16.6/lib/fluent/supervisor.rb:682:in `setup_global_logger'
	10: from /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/fluentd-1.16.6/lib/fluent/config.rb:52:in `build'
	 9: from /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/fluentd-1.16.6/lib/fluent/config.rb:71:in `parse'
	 8: from /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/fluentd-1.16.6/lib/fluent/config/v1_parser.rb:33:in `parse'
	 7: from /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/fluentd-1.16.6/lib/fluent/config/v1_parser.rb:44:in `parse!'
	 6: from /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/fluentd-1.16.6/lib/fluent/config/v1_parser.rb:96:in `parse_element'
	 5: from /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/fluentd-1.16.6/lib/fluent/config/v1_parser.rb:96:in `parse_element'
	 4: from /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/fluentd-1.16.6/lib/fluent/config/v1_parser.rb:131:in `parse_element'
	 3: from /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/fluentd-1.16.6/lib/fluent/config/literal_parser.rb:75:in `parse_literal'
	 2: from /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/fluentd-1.16.6/lib/fluent/config/literal_parser.rb:82:in `scan_string'
	 1: from /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/fluentd-1.16.6/lib/fluent/config/literal_parser.rb:114:in `scan_double_quoted_string'
/Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/fluentd-1.16.6/lib/fluent/config/basic_parser.rb:92:in `parse_error!': unexpected end of file in a double quoted string at fluent.conf line 6,17 (Fluent::ConfigParseError)
  5:   <record>
  6:     hello "world

     -----------------^
  7:

Expected behavior

The string should be parsed like any other multiline string. In this case I'd expect hello to be set to "world\n\n".

Your Environment

- Fluentd version: 1.16.6

Your Configuration

<filter test>
  @type record_transformer
  <record>
    hello "world

"
  </record>
</filter>

Your Error Log

Traceback (most recent call last):
	18: from /Users/me/.rbenv/versions/2.5.1/bin/fluentd:23:in `<main>'
	17: from /Users/me/.rbenv/versions/2.5.1/bin/fluentd:23:in `load'
	16: from /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/fluentd-1.16.6/bin/fluentd:15:in `<top (required)>'
	15: from /Users/me/.rbenv/versions/2.5.1/lib/ruby/site_ruby/2.5.0/rubygems/core_ext/kernel_require.rb:85:in `require'
	14: from /Users/me/.rbenv/versions/2.5.1/lib/ruby/site_ruby/2.5.0/rubygems/core_ext/kernel_require.rb:85:in `require'
	13: from /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/fluentd-1.16.6/lib/fluent/command/fluentd.rb:351:in `<top (required)>'
	12: from /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/fluentd-1.16.6/lib/fluent/supervisor.rb:624:in `configure'
	11: from /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/fluentd-1.16.6/lib/fluent/supervisor.rb:682:in `setup_global_logger'
	10: from /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/fluentd-1.16.6/lib/fluent/config.rb:52:in `build'
	 9: from /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/fluentd-1.16.6/lib/fluent/config.rb:71:in `parse'
	 8: from /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/fluentd-1.16.6/lib/fluent/config/v1_parser.rb:33:in `parse'
	 7: from /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/fluentd-1.16.6/lib/fluent/config/v1_parser.rb:44:in `parse!'
	 6: from /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/fluentd-1.16.6/lib/fluent/config/v1_parser.rb:96:in `parse_element'
	 5: from /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/fluentd-1.16.6/lib/fluent/config/v1_parser.rb:96:in `parse_element'
	 4: from /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/fluentd-1.16.6/lib/fluent/config/v1_parser.rb:131:in `parse_element'
	 3: from /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/fluentd-1.16.6/lib/fluent/config/literal_parser.rb:75:in `parse_literal'
	 2: from /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/fluentd-1.16.6/lib/fluent/config/literal_parser.rb:82:in `scan_string'
	 1: from /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/fluentd-1.16.6/lib/fluent/config/literal_parser.rb:114:in `scan_double_quoted_string'
/Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/fluentd-1.16.6/lib/fluent/config/basic_parser.rb:92:in `parse_error!': unexpected end of file in a double quoted string at fluent.conf line 6,17 (Fluent::ConfigParseError)
  5:   <record>
  6:     hello "world

     -----------------^
  7:

Additional context

No response

Alainx277 avatar Oct 29 '24 13:10 Alainx277

This also happens if a line is terminated with a literal backslash:

<filter test>
  @type record_transformer
  <record>
    hello "world \\
test"
  </record>
</filter>

Alainx277 avatar Oct 29 '24 13:10 Alainx277

It seems that the limitation of current implementation. As a workaround, use "\n" explicitly.

kenhys avatar Oct 30 '24 02:10 kenhys

Thanks for your report. It appears this is a bug of Fluentd.

This works.

<filter test.**>
  @type record_transformer
  <record>
    hello "world
"
  </record>
</filter>

However, the following don't work.

<filter test.**>
  @type record_transformer
  <record>
    hello "world

"
  </record>
</filter>

<filter test.**>
  @type record_transformer
  <record>
    hello 'world
'
  </record>
</filter>

<filter test.**>
  @type record_transformer
  <record>
    hello 'world

'
  </record>
</filter>

daipom avatar Nov 01 '24 03:11 daipom

It looks like single-quoted strings don't support multiline. https://docs.fluentd.org/configuration/config-file#multiline-support-for-quoted-string-array-and-hash-values

So, we need to check double quoted strings behavior, as @Alainx277 says.

daipom avatar Nov 01 '24 04:11 daipom