elastic
loading LS might (rarely) fail due JRuby auto-load bug
JRuby, being Ruby, uses RubyGems - a require 'rubygems' happens by default as the runtime starts.
Some of the constants (such as Gem::Specification) are setup to auto-load on demand while also being explicitly required.
This is problematic and known issue for JRuby 9K, it can be triggered when multiple threads try to do smt with an auto-loaded constant which has been explicitly required (sample reproducer at https://github.com/jruby/jruby/issues/6293).
LS does use Gem::Specification as pipelines are being initialized, sample logged failure from LS 7.6.0 :
[ERROR][logstash.agent ] Failed to execute action
{:action=>LogStash::PipelineAction::Create/pipeline_id:a_pipeline,
:exception=>"NameError",
:message=>"uninitialized constant Gem::Specification",
:backtrace=>[
"org/jruby/RubyModule.java:3742:in `const_missing'",
"/usr/share/logstash/logstash-core/lib/logstash/util/plugin_version.rb:25:in `find_version!'",
"/usr/share/logstash/logstash-core/lib/logstash/util/plugin_version.rb:42:in `find_plugin_version!'",
"/usr/share/logstash/logstash-core/lib/logstash/config/mixin.rb:245:in `print_version_notice'",
"/usr/share/logstash/logstash-core/lib/logstash/config/mixin.rb:231:in `validate'",
"/usr/share/logstash/logstash-core/lib/logstash/config/mixin.rb:86:in `config_init'",
"/usr/share/logstash/logstash-core/lib/logstash/inputs/base.rb:60:in `initialize'",
"org/logstash/plugins/PluginFactoryExt.java:258:in `plugin'",
"org/logstash/plugins/PluginFactoryExt.java:120:in `buildInput'",
"org/logstash/execution/JavaBasePipelineExt.java:50:in `initialize'",
"/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:24:in `initialize'",
"/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:36:in `execute'",
"/usr/share/logstash/logstash-core/lib/logstash/agent.rb:325:in `block in converge_state'"
]}
HINT: The issue is rare to run into and users are advised to attempt to (re-)start.
A work-around is expected to be available with the release of RGs 3.1.5 (and 3.2.0), where auto-loaded parts are no longer being explicitly required. A proper fix is expected to be available in JRuby 9.3.
I got the same issue, running multiple pipelines in docker withour restarting the container, the pipeline finished by starting itself I'm using logstash 7.16.2
[2022-04-14T11:00:34,164][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:my_pipeline, :exception=>"Java::JavaLang::IllegalStateException", :message=>"Unable to configure plugins: (NameError) uninitialized constant Gem::Specification", :backtrace=>["org.logstash.config.ir.CompiledPipeline.<init>(CompiledPipeline.java:119)", "org.logstash.execution.JavaBasePipelineExt.initialize(JavaBasePipelineExt.java:86)", "org.logstash.execution.JavaBasePipelineExt$INVOKER$i$1$0$initialize.call(JavaBasePipelineExt$INVOKER$i$1$0$initialize.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:837)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuper(IRRuntimeHelpers.java:1169)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuperSplatArgs(IRRuntimeHelpers.java:1156)", "org.jruby.ir.targets.InstanceSuperInvokeSite.invoke(InstanceSuperInvokeSite.java:39)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$initialize$0(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:47)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:80)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:333)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:87)", "org.jruby.RubyClass.newInstance(RubyClass.java:939)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(RubyClass$INVOKER$i$newInstance.gen)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:207)", "usr.share.logstash.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0(/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:52)", "usr.share.logstash.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0$__VARARGS__(/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:50)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:80)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:207)", "usr.share.logstash.logstash_minus_core.lib.logstash.agent.RUBY$block$converge_state$2(/usr/share/logstash/logstash-core/lib/logstash/agent.rb:383)", "org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:138)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:52)", "org.jruby.runtime.Block.call(Block.java:139)", "org.jruby.RubyProc.call(RubyProc.java:318)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:105)", "java.base/java.lang.Thread.run(Thread.java:829)"]}
warning: thread "Converge PipelineAction::Create<my_pipeline>" terminated with exception (report_on_exception is true):
LogStash::Error: Don't know how to handle `Java::JavaLang::IllegalStateException` for `PipelineAction::Create<my_pipeline>`
create at org/logstash/execution/ConvergeResultExt.java:135
add at org/logstash/execution/ConvergeResultExt.java:60
converge_state at /usr/share/logstash/logstash-core/lib/logstash/agent.rb:396
[2022-04-14T11:00:34,168][ERROR][logstash.agent ] Failed to execute action
the JRuby 9.3 upgrade was shipped at https://github.com/elastic/logstash/pull/14114 (we expect this issue to be resolved in Logstash >= 8.3.0)
Closing this issue as the related work is now complete. This will be available in 8.4
