logstash feat(syntax): allow comments in hashes and before EOF

Release notes

Fixed the syntax rules to allow comments between hash entries and immediately before the end of a file. Comments in these places would previously give syntax errors.

What does this PR do?

In the grammer definitions for hashes, whitespace was replaced with cs to allow either whitespace or comments. Additionally, the grammar definition for comments was previously required to end with a newline, now it can end with a newline or EOF, using the "not anything" treetop rule !..

Why is it important/What is the impact to the user?

Users get surprised when they find out that comments are not allowed in hashes or at the end of a file. This is allowed in practically all other languages, including ruby. This PR thus aligns Logstash syntax with user expectations.

Checklist

[x] My code follows the style guidelines of this project
- AFAICT, no automated check for this? I believe it follows the guide though.
[x] I have commented my code, particularly in hard-to-understand areas
- I only added syntax, generated code (treetop), and test code. Aside from the generated code, changes are trivial.
[x] I have made corresponding changes to the documentation
- No docs changes should be needed.
[x] I have made corresponding change to the default configuration files (and/or docker env variables)
- Should also not be needed.
[x] I have added tests that prove my fix is effective or that my feature works
- Yes though!

Author's Checklist

[ ]

How to test this PR locally

To run the test that I added, execute SPEC_OPTS="-fd -P logstash-core/spec/logstash/config/config_ast_spec.rb" ./gradlew :logstash-core:rubyTests --tests org.logstash.RSpecTests (this command is copied from README.md but with the path changed).

Alternatively, create a hash in a logstash configuration manually with at least two entries, and put a comment between them. It will work on the patched logstash, and give a syntax error on the unpatched logstash.

Related issues

Closes #15651
Closes #15561

Use cases

For example,

filter {
  mutate {
    add_field => {
      "field1" => "value1"
      # You can now put a comment between hash entries, like here
      "field2" => "value"
    }
  }
}
# You can now also put a comment before an EOF (i.e. no trailing newline)

Screenshots

Logs

Apr 05 '24 13:04 jonaslb

Hi @andsel, kindly requesting a review from you since you've worked on the syntax before. Hope you can find time for it? :)

Apr 18 '24 10:04 jonaslb

@jonaslb thank's a lot for this contribution. Left a couple of comments.

Tried with the following:
input {
  stdin {
    codec => json {
      target => "[document]"
      # this comment already worked
      ecs_compatibility => disabled
    }
  }
}

The reason that comment already worked is because this isn't a Hash, but arguments for a plugin, which happens to look essentially the same but they're different things in the grammar (minor differences in allowed value types iirc).

Apr 22 '24 14:04 jonaslb

The reason that comment already worked is because this isn't a Hash, but arguments for a plugin, which happens to look essentially the same but they're different things in the grammar (minor differences in allowed value types iirc).

Yes you are right, is something different. 👍

Apr 22 '24 14:04 andsel

Hi again @andsel , to be clear about what the pattern matches, I created a rule with a telling name for it (and used it in the comment rule):

  rule newline_or_eoi
    # `!.` is a negative lookahead for 'anything', i.e. it matches at the end of input.
    ("\r"? "\n") / !.
  end

What do you think of this?

Apr 30 '24 10:04 jonaslb

BK run https://buildkite.com/elastic/logstash-pull-request-pipeline/builds/1061

May 02 '24 12:05 andsel

Nothing seems to have been triggered by the "BK run" comment or the results are not visible here? Or - maybe it is what's on the link - but it's all green?

May 08 '24 07:05 jonaslb

buildkite test this