[ and ] in field names still cause parse failures

[garethhumphriesgkc]

Ref: Logstash bug 14821

Logstash information:

$ docker run --rm -p 6996:6996 docker.elastic.co/logstash/logstash:8.5.3 -e '' --version
Using bundled JDK: /usr/share/logstash/jdk
Sending Logstash logs to /usr/share/logstash/logs which is now configured via log4j2.properties
[2022-12-19T20:20:49,552][INFO ][logstash.runner          ] Log4j configuration path used is: /usr/share/logstash/config/log4j2.properties
logstash 8.5.3
jruby 9.3.9.0 (2.6.8) 2022-10-24 537cd1f8bc OpenJDK 64-Bit Server VM 17.0.5+8 on 17.0.5+8 +indy +jit [x86_64-linux]
java 17.0.5 (Eclipse Adoptium)
jvm OpenJDK 64-Bit Server VM / 17.0.5+8

OS version (uname -a if on a Unix-like system):

$ uname -a ; docker --version
Linux workstation 5.19.0-26-generic #27-Ubuntu SMP PREEMPT_DYNAMIC Wed Nov 23 20:44:15 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Docker version 20.10.16, build 20.10.16-0ubuntu1

Description of the problem including expected versus actual behaviour: I raised issue 13606 around the failure of logstash to handle incoming records with embedded ['s in field names. It was closed under PR 14044, but I tested it today with 8.5.3 and the original repro still works:

When sending csv data though the csv filter or input codec, if one of the field names contains a specials char (e.g. '['), the parser fails.

Steps to reproduce:

Start a very simple logstash instance:

$ docker run --rm -p 6996:6996 docker.elastic.co/logstash/logstash:8.5.3 -e 'input {tcp { port => 6996 } }filter { csv{autodetect_column_names => true }}output { stdout {} }' --field-reference-escape-style ampersand

Send in some sample data - when there are special chars present _jsonparsefailure occurs:

echo -e 'field[name],value\nstr[substr],true' | nc -w1 127.0.0.1 6996

[2022-12-19T20:40:59,187][WARN ][logstash.filters.csv     ][main][128ac161fdf34f40fbbcb2d3638ef1f2a3ddef55622bee33ab4926a03304276f] Error parsing csv {:field=>"message", :source=>"field[name],value", :exception=>#<RuntimeError: Invalid FieldReference: `[str[substr]]`>}
{
         "event" => {
        "original" => "field[name],value"
    },
       "message" => "field[name],value",
    "@timestamp" => 2022-12-19T20:40:58.959832635Z,
      "@version" => "1",
          "tags" => [
        [0] "_csvparsefailure"
    ]
}