ArrayIndexOutOfBoundsException: Index -1 out of bounds for length 80 while PipelineAction::Create

[sasikiranvaddi]

We have two pipelines configured, "opensearch" and "logstash". During the startup of logstash process OpenSearch pipeline has been created and while converging Logstash pipeline the logstash is failing with ArrayIndexOutOfBoundsException.

The exception has occurred while processing the ThreadContext. at org.jruby.runtime.ThreadContext.popFrameReal(ThreadContext.java:570)

To recover from it, we produced a graceful restart of logstash process, after which we could see the pipelines running successfuly. [logstash.agent] Pipelines running {:count=>2, :running_pipelines=>[:opensearch, :logstash], :non_running_pipelines=>[]}

Versions:

Logstash: 8.13.0
Jruby: 9.4.5.0
Java: JDK11

[logstash.runner] Starting Logstash {'logstash.version'=>'8.13.0', 'jruby.version'=>'jruby 9.4.5.0 (3.1.4) 2023-11-02 1abae2700f OpenJDK 64-Bit Server VM 11.0.23+9-suse-150000.3.113.1-x8664 on 11.0.23+9-suse-150000.3.113.1-x8664 +indy +jit [x86_64-linux]'}

    pipelines.yml: 
      - pipeline.id: logstash
        queue.type: persisted
        queue.max_bytes: 1024mb
        path.config: "/opt/logstash/resource/logstash.conf"
      - pipeline.id: opensearch
        queue.type: persisted
        queue.max_bytes: 1024mb
        path.config: "/opt/logstash/resource/searchengine.conf"

On a side note, there is an issue w.r.t metric, the logstash recommended to log an issue. Please let me know if it requires a separate ticket.

[org.logstash.instrument.metrics.gauge.LazyDelegatingGauge] A gauge metric of an unknown type (org.jruby.specialized.RubyArrayOneObject) has been created for key: send_to. This may result in invalid serialization. It is recommended to log an issue to the responsible developer/development team.
2024-07-03 20:40:38,246 pool-5-thread-1 ERROR An exception occurred processing Appender plain_console org.jruby.exceptions.TypeError: (TypeError) no implicit conversion of Hash into String
at org.jruby.RubyKernel.inspect(org/jruby/RubyKernel.java:2241)
at org.jruby.RubyHash.inspect(org/jruby/RubyHash.java:945)
at org.jruby.RubyKernel.inspect(org/jruby/RubyKernel.java:2241)
at org.jruby.RubyKernel.inspect(org/jruby/RubyKernel.java:2241)
at org.jruby.RubyKernel.inspect(org/jruby/RubyKernel.java:2241)
at org.jruby.RubyKernel.inspect(org/jruby/RubyKernel.java:2241)
at org.jruby.RubyHash.inspect(org/jruby/RubyHash.java:945)
at org.jruby.RubyHash.to_s(org/jruby/RubyHash.java:1011)
at org.logstash.log.LoggerExt.error(org/logstash/log/LoggerExt.java:127)
at RUBY.update(/opt/logstash/logstash-core/lib/logstash/instrument/periodic_poller/base.rb:53)
at RUBY.notify_to(/opt/logstash/vendor/bundle/jruby/3.1.0/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/collection/copy_on_notify_observer_set.rb:102)
at org.jruby.RubyHash.each(org/jruby/RubyHash.java:1601)
at RUBY.notify_to(/opt/logstash/vendor/bundle/jruby/3.1.0/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/collection/copy_on_notify_observer_set.rb:100)
at RUBY.notify_observers(/opt/logstash/vendor/bundle/jruby/3.1.0/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/collection/copy_on_notify_observer_set.rb:64)
at RUBY.execute_task(/opt/logstash/vendor/bundle/jruby/3.1.0/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/timer_task.rb:317)
at RUBY.execute(/opt/logstash/vendor/bundle/jruby/3.1.0/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/executor/safe_task_executor.rb:24)
at RUBY.execute(/opt/logstash/vendor/bundle/jruby/3.1.0/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/executor/safe_task_executor.rb:19)
at RUBY.safe_execute(/opt/logstash/vendor/bundle/jruby/3.1.0/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/ivar.rb:169)
at RUBY.process_task(/opt/logstash/vendor/bundle/jruby/3.1.0/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/scheduled_task.rb:285)
at RUBY.process_tasks(/opt/logstash/vendor/bundle/jruby/3.1.0/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/executor/timer_set.rb:165)
at RUBY.run(/opt/logstash/vendor/bundle/jruby/3.1.0/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/executor/java_executor_service.rb:79)

Please find the attached logs. ArrayOutOfBoundsandMetricException.txt