logstash icon indicating copy to clipboard operation
logstash copied to clipboard
verified publisher icon elastic

Update the java plugin loading logic to process the manifest file.

Open andsel opened this issue 3 years ago • 1 comments

Release notes

Enable the usage of not shadowed Java plugins

What does this PR do?

Update the Java plugins loading from gems to use multijar plugins. Before this PR the example Java plugins come with a single shadowed jar with all the dependent jars squashed into. The shadowed jar also bundle Logstash classes and logging libraries, driving to an increase of file size, but also polluting the classpath. With PR https://github.com/logstash-plugins/logstash-input-java_input_example/pull/15 it fixes the problem on plugin side. Logstash needs to recognize and properly load multijar plugins. This PR list all jars bundled in the gem pack, looking for the MANIFEST marked jar file that contains the plugin implementation class.

Why is it important/What is the impact to the user?

Let the developer to implement pure Java Logstash plugins that could be used into the project.

The native Java plugins generally bundles all the the transitive dependencies inside an uber jar. However, due to the Gradle's shadow plugin, this generates a problem because the uber jar contains also Logstash's core classes, and this could generate a problem with class loading. On the other side, mixed Ruby/Java plugins can ship multiple jars used by the plugins itself. This PR aims to bundle multiple jars inside a native Java plugins, permitting the correct loading of the plugin's main class through the usage of manifest file.

Checklist

  • [x] My code follows the style guidelines of this project
  • [x] I have commented my code, particularly in hard-to-understand areas
  • [ ] I have made corresponding changes to the documentation
  • [ ] I have made corresponding change to the default configuration files (and/or docker env variables)
  • [ ] I have added tests that prove my fix is effective or that my feature works

Author's Checklist

  • [ ]

How to test this PR locally

Test as described in PR https://github.com/logstash-plugins/logstash-input-java_input_example/pull/15

Related issues

  • Relates https://github.com/logstash-plugins/logstash-input-java_input_example/pull/15

Use cases

As a developer I want to bundle only the required libraries used by a native plugin, bundling them and permitting the correct loading of the plugin's main class.

Screenshots

Logs

Error happening when loading a Java plugin with multuple jars contained:

[FATAL] 2022-08-29 09:40:26.675 [main] Logstash - Logstash stopped processing because of an error: (LoadError) Java plugin 'logstash-input-java_input_example' does not contain a single jar file with the plugin's name and version
org.jruby.exceptions.LoadError: (LoadError) Java plugin 'logstash-input-java_input_example' does not contain a single jar file with the plugin's name and version
	at home.andrea.workspace.logstash_andsel.logstash_minus_core.lib.logstash.plugins.registry.load_available_plugins(/home/andrea/workspace/logstash_andsel/logstash-core/lib/logstash/plugins/registry.rb:159) ~[?:?]
	at org.jruby.RubyArray.each(org/jruby/RubyArray.java:1865) ~[jruby.jar:?]
	at home.andrea.workspace.logstash_andsel.logstash_minus_core.lib.logstash.plugins.registry.load_available_plugins(/home/andrea/workspace/logstash_andsel/logstash-core/lib/logstash/plugins/registry.rb:153) ~[?:?]
	at home.andrea.workspace.logstash_andsel.logstash_minus_core.lib.logstash.plugins.registry.setup!(/home/andrea/workspace/logstash_andsel/logstash-core/lib/logstash/plugins/registry.rb:131) ~[?:?]
	at RUBY.<class:Runner>(/home/andrea/workspace/logstash_andsel/logstash-core/lib/logstash/runner.rb:262) ~[?:?]
	at RUBY.<main>(/home/andrea/workspace/logstash_andsel/logstash-core/lib/logstash/runner.rb:55) ~[?:?]
	at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:1017) ~[jruby.jar:?]
	at home.andrea.workspace.logstash_andsel.lib.bootstrap.environment.<main>(/home/andrea/workspace/logstash_andsel/lib/bootstrap/environment.rb:89) ~[?:?]

andsel avatar Apr 07 '22 12:04 andsel

Jenkins test this please

andsel avatar Apr 07 '22 14:04 andsel

The CI fails is on integration test:

./gradlew clean runIntegrationTests -PrubyIntegrationSpecs="specs/install_java_plugin_spec.rb"

with the error:

    Starting Logstash: /home/andrea/workspace/logstash_andsel/build/logstash-8.7.0-SNAPSHOT/bin/logstash ["-f", "/tmp/studtmp-cdbbf93353cd54233dc1f5b5673a7c672d9b3325dca6fbc098596664f4a3", "--path.settings", "/tmp/studtmp-0fcba89b07f26b3d5f27f2ab6f8f4d490f8609192c84228c1a6dfa19532b"] (pwd: /home/andrea/workspace/logstash_andsel/qa/integration)
    Logstash started with PID 190329, using java: bundled java
    Using bundled JDK: /home/andrea/workspace/logstash_andsel/build/logstash-8.7.0-SNAPSHOT/jdk
    [FATAL] 2022-12-01 18:59:43.603 [main] Logstash - Logstash stopped processing because of an error: (NoMethodError) undefined method `relative_path' for nil:NilClass
    org.jruby.exceptions.NoMethodError: (NoMethodError) undefined method `relative_path' for nil:NilClass
        at RUBY.find_plugin_jar(/home/andrea/workspace/logstash_andsel/build/logstash-8.7.0-SNAPSHOT/logstash-core/lib/logstash/plugins/registry.rb:176) ~[?:?]
        at home.andrea.workspace.logstash_andsel.build.logstash_minus_8_dot_7_dot_0_minus_SNAPSHOT.logstash_minus_core.lib.logstash.plugins.registry.load_available_plugins(/home/andrea/workspace/logstash_andsel/build/logstash-8.7.0-SNAPSHOT/logstash-core/lib/logstash/plugins/registry.rb:184) ~[?:?]
        at org.jruby.RubyArray.each(org/jruby/RubyArray.java:1865) ~[jruby.jar:?]
        at RUBY.load_available_plugins(/home/andrea/workspace/logstash_andsel/build/logstash-8.7.0-SNAPSHOT/logstash-core/lib/logstash/plugins/registry.rb:182) ~[?:?]
        at RUBY.setup!(/home/andrea/workspace/logstash_andsel/build/logstash-8.7.0-SNAPSHOT/logstash-core/lib/logstash/plugins/registry.rb:131) ~[?:?]
        at RUBY.<class:Runner>(/home/andrea/workspace/logstash_andsel/build/logstash-8.7.0-SNAPSHOT/logstash-core/lib/logstash/runner.rb:262) ~[?:?]
        at RUBY.<main>(/home/andrea/workspace/logstash_andsel/build/logstash-8.7.0-SNAPSHOT/logstash-core/lib/logstash/runner.rb:55) ~[?:?]
        at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:1017) ~[jruby.jar:?]
        at home.andrea.workspace.logstash_andsel.build.logstash_minus_8_dot_7_dot_0_minus_SNAPSHOT.lib.bootstrap.environment.<main>(/home/andrea/workspace/logstash_andsel/build/logstash-8.7.0-SNAPSHOT/lib/bootstrap/environment.rb:89) ~[?:?]

andsel avatar Dec 01 '22 19:12 andsel