logstash "[" key in an event crashes logstash

For all general issues, please provide the following details for fast resolution:

Version: 7.4.0
Operating System: Linux
Config File (if you have sensitive info, please remove it): Just read it with any input plugin, I encountered this while reading with elastic search input plugin.
Sample Data: {"[": null}
Steps to Reproduce: Try creating an event object with the sample data pasted above. It fails with the following error, Error: Invalid FieldReference: [ Exception: Java::OrgLogstash::FieldReference::IllegalSyntaxException Stack: org.logstash.FieldReference$StrictTokenizer.tokenize(FieldReference.java:283)

As we don't have control on the incoming data, should logstash fail when it fails to parse the incoming data? Although it may look fair from the logstash perspective, this crash resets the elastic input plugin which then starts reading from the start again. One bad record results in an infinite loop! The fix I believe should come from logstash instead of elastic input plugin as this potentially can affect multiple other plugins as well. At least there should be a way to ignore such data errors while creating events at the logstash level.

Jul 19 '20 10:07 sribharghava

Can you provide a backtrace? I believe we have resolved a few of these since 7.4 shipped, but I'd be glad to help chase this instance down. Many plugins/codecs will catch this kind of exception and produce a tagged event that contains a serialized form of the payload.

Jul 20 '20 16:07 yaauie

Error: Invalid FieldReference: [Exception: Java::OrgLogstash::FieldReference::IllegalSyntaxException Stack: org.logstash.FieldReference$StrictTokenizer.tokenize(FieldReference.java:283) org.logstash.FieldReference.parse(FieldReference.java:184) org.logstash.FieldReference.parseToCache(FieldReference.java:175) org.logstash.FieldReference.from(FieldReference.java:107) org.logstash.FieldReference.lambda$from$0(FieldReference.java:98) java.base/java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1737) org.logstash.FieldReference.from(FieldReference.java:98) org.logstash.ConvertedMap.convertKey(ConvertedMap.java:102) org.logstash.ConvertedMap.access$000(ConvertedMap.java:24) org.logstash.ConvertedMap$1.visit(ConvertedMap.java:35) org.logstash.ConvertedMap$1.visit(ConvertedMap.java:29) org.jruby.RubyHash.visitLimited(RubyHash.java:689) org.jruby.RubyHash.visitAll(RubyHash.java:674) org.logstash.ConvertedMap.newFromRubyHash(ConvertedMap.java:69) org.logstash.ConvertedMap.newFromRubyHash(ConvertedMap.java:64) org.logstash.Valuefier.lambda$initConverters$12(Valuefier.java:151) org.logstash.Valuefier.convert(Valuefier.java:74) org.logstash.ConvertedMap$1.visit(ConvertedMap.java:35) org.logstash.ConvertedMap$1.visit(ConvertedMap.java:29) org.jruby.RubyHash.visitLimited(RubyHash.java:689) org.jruby.RubyHash.visitAll(RubyHash.java:674) org.logstash.ConvertedMap.newFromRubyHash(ConvertedMap.java:69) org.logstash.ConvertedMap.newFromRubyHash(ConvertedMap.java:64) org.logstash.Valuefier.lambda$initConverters$12(Valuefier.java:151) org.logstash.Valuefier.convert(Valuefier.java:74) org.logstash.ConvertedMap$1.visit(ConvertedMap.java:35) org.logstash.ConvertedMap$1.visit(ConvertedMap.java:29) org.jruby.RubyHash.visitLimited(RubyHash.java:689) org.jruby.RubyHash.visitAll(RubyHash.java:674) org.logstash.ConvertedMap.newFromRubyHash(ConvertedMap.java:69) org.logstash.ConvertedMap.newFromRubyHash(ConvertedMap.java:64) org.logstash.Valuefier.lambda$initConverters$12(Valuefier.java:151) org.logstash.Valuefier.convert(Valuefier.java:74) org.logstash.ConvertedMap$1.visit(ConvertedMap.java:35) org.logstash.ConvertedMap$1.visit(ConvertedMap.java:29) org.jruby.RubyHash.visitLimited(RubyHash.java:689) org.jruby.RubyHash.visitAll(RubyHash.java:674) org.logstash.ConvertedMap.newFromRubyHash(ConvertedMap.java:69) org.logstash.ext.JrubyEventExtLibrary$RubyEvent.ruby_initialize(JrubyEventExtLibrary.java:72) usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_elasticsearch_minus_4_dot_0_dot_2.lib.logstash.inputs.elasticsearch.RUBY$method$push_hit$0(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-elasticsearch-4.0.2/lib/logstash/inputs/elasticsearch.rb:162) usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_elasticsearch_minus_4_dot_0_dot_2.lib.logstash.inputs.elasticsearch.RUBY$block$run$1(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-elasticsearch-4.0.2/lib/logstash/inputs/elasticsearch.rb:144) org.jruby.runtime.CompiledIRBlockBody.yieldDirect(CompiledIRBlockBody.java:146) org.jruby.runtime.BlockBody.yield(BlockBody.java:114) org.jruby.runtime.Block.yield(Block.java:170) org.jruby.RubyArray.each(RubyArray.java:1800) org.jruby.RubyArray$INVOKER$i$0$0$each.call(RubyArray$INVOKER$i$0$0$each.gen) org.jruby.internal.runtime.methods.JavaMethod$JavaMethodZeroBlock.call(JavaMethod.java:555) org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:177) usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_elasticsearch_minus_4_dot_0_dot_2.lib.logstash.inputs.elasticsearch.RUBY$method$run$0(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-elasticsearch-4.0.2/lib/logstash/inputs/elasticsearch.rb:144) usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_elasticsearch_minus_4_dot_0_dot_2.lib.logstash.inputs.elasticsearch.RUBY$method$run$0$__VARARGS__(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-elasticsearch-4.0.2/lib/logstash/inputs/elasticsearch.rb) org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:91) org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:90) org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:183) usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$inputworker$0(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:314) usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$inputworker$0$__VARARGS__(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb) org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:91) org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:90) org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:183) usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$block$start_input$1(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:306) org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:136) org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:77) org.jruby.runtime.Block.call(Block.java:129) org.jruby.RubyProc.call(RubyProc.java:295) org.jruby.RubyProc.call(RubyProc.java:274) org.jruby.RubyProc.call(RubyProc.java:270) org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:105) java.base/java.lang.Thread.run(Thread.java:834)

Jul 20 '20 17:07 sribharghava

I also have this situation, looking forward to solving it. My question

Jul 21 '20 09:07 Zoom2016

Any updates on this? This seems to still be an issue with 7.12.1

May 31 '21 10:05 reneschroeder0000

I resolved it by stringifying the incoming json document in the respective plug-in (couchdb_changes.rb). Works fine. @Zoom2016 @reneschroeder0000

Nov 24 '21 11:11 fahadmaq007

any updates on this? this bug is causing our pipelines to crash and we have no way to fix this

Jun 27 '22 17:06 balusarakesh

any updates on this?

Oct 19 '22 13:10 abdelbarre

This was resolved in 8.3.0 with #14044.

Even with the above fix in place, the Elasticsearch input plugin should certainly be more resilient to failure in general, so I have filed https://github.com/logstash-plugins/logstash-input-elasticsearch/issues/182

Oct 19 '22 14:10 yaauie