Install specific version of plugin fails

[suyograo]

plugin install --version fails

From: https://discuss.elastic.co/t/install-specific-plugin-version/1407

sudo /opt/logstash/bin/plugin install --version 0.2.5 logstash-output-elasticsearch
Validating logstash-output-elasticsearch-0.2.5
Installing logstash-output-elasticsearch
Plugin not found, aborting
ERROR: Installation Aborted, message: You have requested:
  logstash-output-elasticsearch = 0.2.5

The bundle currently has logstash-output-elasticsearch locked at 0.2.4.
Try running `bundle update logstash-output-elasticsearch`

[purbon]

The behaviour in here is reasonable, you can not install a new version as it, you should update it. Do you expect this call to install anyway?

At less I think we should update the wording in here so the message is clear to the end user.

[ralphm]

This seems to be related to #3152.

I think I'd like install to just install the specified version, and update dependencies if needed, possibly after confirmation (or a --force). It seems that plugin update lacks a version flag, so it will always try to install the latest known version. Additionally, the chef cookbook by lusis doesn't (currently) have support for update.

[AdeMiller]

It seems that plugin update lacks a version flag, so it will always try to install the latest known version.

This is horrible. Logstash is pretty unstable as it is. Installing the latest (and possibly not) greatest version seems like a bad idea. In addition this means that there is no way to ensure version compatibility across different installs. For example a deployment one week in PROD may end up with a newer plugin that the deployment tested in QA the previous day.

[purbon]

@AdeMiller I don't follow your issue, i would really like to understand the source of your pain. If you install LS it comes with a set of bundler plugins, this are the same always unless you change them manually. If you update an installation to the latest versions and not another, then is not a logstash issue I think. Looking forward to understand more your issue.

[AdeMiller]

The scenario is simple. I install a specific version of LS and some additional plugins with specific versions. At some later date I decide to update a plugin to a newer version. I need to do this in more than one environment QA/DEV/PROD etc. I want to ensure that the version I upgrade to in QA and test is the same version I subsequently deploy in the other environments. Upgrade simply gets the latest so upgrading in QA on Monday and PROD on Wednesday would lead to the possibility that I get two (different) latest versions.

Right now the workaround for this is to uninstall each plugin if the installed version is different from the required version (which has it's own issues) and then install the specific version.

All of the above is automated (through) Ansible so upgrade latest is incompatible with a system that assumes operations are idempotent.

[purbon]

@AdeMiller yeah, in this case this is painful I understand. If you want to control the version of your plugins I propose your for now a few strategies:

• Use the offline functionality we recently merged see (https://github.com/elastic/logstash/pull/3404), this is going to be release soon with the next minor release. This will let you dump a set of plugins, you know and like, including dependencies, and make them available to different machines. not perfect, but might be a solution.
• You can also have your own gem server, where there are only available the plugin versions you know and like, then using this custom server your LS installation will only get the approved plugins.

Does this helps you?

/purbon

[tanderegg]

@purbon @AdeMiller I've run into this problem as well, it would be great to get an idempotent upgrade option for /bin/plugin upgrade (or just install). It would be very helpful for those who use Ansible or similar tools. The gem server option is a creative idea, although it adds more complexity. I might look into that in the mean time.

[mcandre]

+1

[ph]

It still a problem with 5.X?

[wongchao]

+1 kafka 0.9 logstash 5.1.2 , by default, logstash-output-kafka is v5.

bin/logstash-plugin install --version 4.0.1 logstash-output-kafka

[PhaedrusTheGreek]

+1 for a fix here.

As a workaround, a constraint can be added to the file Gemfile in the Logstash home directory:

e.g., to install logstash-input-tcp 4.2.4 (not 5.0.2) in Logstash 5.3.0,

1) Edit the file called "Gemfile" in the logstash home directory and change this line:

gem "logstash-input-tcp"

to this:

gem "logstash-input-tcp", "< 5"

2) Update the plugin:

bin/logstash-plugin update logstash-input-tcp

Updating logstash-input-tcp
Updated logstash-codec-json 3.0.2 to 3.0.4
Updated logstash-codec-json_lines 3.0.2 to 3.0.4
Updated logstash-codec-line 3.0.2 to 3.0.4
Updated logstash-codec-plain 3.0.2 to 3.0.4
Updated logstash-input-tcp 4.1.0 to 4.2.4

[jakelandis]

Is this still a problem with 5.x ?

I just tested the following using a freshly unzipped 5.5.0 release

logstash-5.5.0 $ find vendor -name logstash-output-kafka*
vendor/bundle/jruby/1.9/gems/logstash-output-kafka-5.1.7
vendor/bundle/jruby/1.9/gems/logstash-output-kafka-5.1.7/lib/logstash-output-kafka_jars.rb
vendor/bundle/jruby/1.9/gems/logstash-output-kafka-5.1.7/logstash-output-kafka.gemspec
vendor/bundle/jruby/1.9/specifications/logstash-output-kafka-5.1.7.gemspec

logstash-5.5.0 $ bin/logstash-plugin install --version 4.0.1 logstash-output-kafka
Validating logstash-output-kafka-4.0.1
Installing logstash-output-kafka
Installation successful

logstash-5.5.0 $ find vendor -name logstash-output-kafka*
vendor/bundle/jruby/1.9/cache/logstash-output-kafka-4.0.1.gem
vendor/bundle/jruby/1.9/gems/logstash-output-kafka-4.0.1
vendor/bundle/jruby/1.9/gems/logstash-output-kafka-4.0.1/lib/logstash-output-kafka_jars.rb
vendor/bundle/jruby/1.9/gems/logstash-output-kafka-4.0.1/logstash-output-kafka.gemspec
vendor/bundle/jruby/1.9/gems/logstash-output-kafka-5.1.7
vendor/bundle/jruby/1.9/gems/logstash-output-kafka-5.1.7/lib/logstash-output-kafka_jars.rb
vendor/bundle/jruby/1.9/gems/logstash-output-kafka-5.1.7/logstash-output-kafka.gemspec
vendor/bundle/jruby/1.9/specifications/logstash-output-kafka-4.0.1.gemspec
vendor/bundle/jruby/1.9/specifications/logstash-output-kafka-5.1.7.gemspec

logstash-5.5.0 $ grep logstash-output-kafka Gemfile
gem "logstash-output-kafka", "4.0.1"

logstash-5.5.0 $ grep logstash-output-kafka* Gemfile.jruby-1.9.lock 
    logstash-output-kafka (4.0.1)
  logstash-output-kafka (= 4.0.1)

Note - I did not actually run it with this configuration.

[PhaedrusTheGreek]

Seeing this in 5.5.2:

$  bin/logstash-plugin install --version 4.2.4 logstash-input-tcp
Validating logstash-input-tcp-4.2.4
Installing logstash-input-tcp
Plugin not found, aborting
ERROR: Installation Aborted, message: You have requested:
  logstash-input-tcp = 4.2.4

The bundle currently has logstash-input-tcp locked at 4.1.2.
Try running `bundle update logstash-input-tcp`

[guyboertje]

@ph What about making bin/logstash-plugin update --version 4.0.1 logstash-output-kafka actually do a uninstall then an install behind the scenes (with rollback on error, obviously)?

[ph]

@guyboertje I think its a pretty good idea.

[MorrieAtElastic]

I have another customer encountering problems installing V4.1.4 of the s3 output plugin against his 5.6.4 logstash. While the "install" option failed due to gem version conflicts, the "update" option successfully installed the plugin.

I'd like to see this problem finally resolved one way or another: either document the differences in functionality between the install and update options of "logstash-plugin"; fix the "install" option to handle associated library updates; or add a --version option to the update function (my personal choice). But do something to close out this problem which is now >= 3 years old.

[geekpete]

I tried this just now with some recent versions, my example is downgrading Beats input to an older plugin version in 8.x Logstash for some testing/isolation purposes:

./logstash-8.6.2/bin/logstash-plugin list --verbose | rg beats
logstash-input-beats (6.4.4)

./logstash-8.6.2/bin/logstash-plugin install --version=6.2.6 logstash-input-beats
Using bundled JDK: /home/peter/Apps/logstash-8.6.2/jdk
Validating logstash-input-beats-6.2.6
Resolving mixin dependencies
Updating mixin dependencies logstash-mixin-ecs_compatibility_support, logstash-mixin-event_support
Bundler attempted to update logstash-mixin-ecs_compatibility_support but its version stayed the same
Bundler attempted to update logstash-mixin-event_support but its version stayed the same
Installing logstash-input-beats
Installation successful

./logstash-8.6.2/bin/logstash-plugin list --verbose | rg beats
logstash-input-beats (6.2.6)

to use the original issue example, switching the Elasticsearch output plugin...

Check plugin versions bundled with different stack versions:

./logstash-8.6.2/bin/logstash-plugin list --verbose | rg logstash-output-elasticsearch
logstash-output-elasticsearch (11.12.4)

 ./logstash-7.17.14/bin/logstash-plugin list --verbose | rg logstash-output-elasticsearch
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
logstash-output-elasticsearch (11.4.1)

Test downgrade to older plugin version in newer stack version:

./logstash-8.6.2/bin/logstash-plugin install --version=11.4.1 logstash-output-elasticsearch
Using bundled JDK: /home/peter/Apps/logstash-8.6.2/jdk
Validating logstash-output-elasticsearch-11.4.1
Resolving mixin dependencies
Updating mixin dependencies logstash-mixin-ecs_compatibility_support, logstash-mixin-deprecation_logger_support, logstash-mixin-ca_trusted_fingerprint_support
Bundler attempted to update logstash-mixin-ecs_compatibility_support but its version stayed the same
Bundler attempted to update logstash-mixin-deprecation_logger_support but its version stayed the same
Bundler attempted to update logstash-mixin-ca_trusted_fingerprint_support but its version stayed the same
Installing logstash-output-elasticsearch
Installation successful

./logstash-8.6.2/bin/logstash-plugin list --verbose | rg logstash-output-elasticsearch
logstash-output-elasticsearch (11.4.1)

Test upgrade to newer plugin version in older stack version:

./logstash-7.17.14/bin/logstash-plugin install --version=11.12.4 logstash-output-elasticsearch
Using bundled JDK: /home/peter/Apps/logstash-7.17.14/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Validating logstash-output-elasticsearch-11.12.4
Resolving mixin dependencies
Updating mixin dependencies logstash-mixin-ecs_compatibility_support, logstash-mixin-deprecation_logger_support
Bundler attempted to update logstash-mixin-ecs_compatibility_support but its version stayed the same
Bundler attempted to update logstash-mixin-deprecation_logger_support but its version stayed the same
Installing logstash-output-elasticsearch
Installation successful

./logstash-7.17.14/bin/logstash-plugin list --verbose | rg logstash-output-elasticsearch
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
logstash-output-elasticsearch (11.12.4)

So it seems the dependency handling is now more graceful for downgrade/upgrade (for the plugins tested in these examples anyway) ?

Does this mean the original issue is solved?