elastic
Address backward-compatibility of plugins using new Deprecation Logger
Plugins will be unable to easily take advantage of the deprecation logger introduced in https://github.com/elastic/logstash/pull/11260, because introducing it to a plugin means that plugin will no longer work on Logstashes < 7.6.0.
We rely fairly heavily on the ability to release fixes in plugins that can be used in older Logstashes, and forcing major-version branching on any version that wants to use this deprecation logger would introduce a lot of overhead in terms of backports.
Ruby Plugin API
On the Ruby API, a plugin can use Kernel#defined? to use the deprecation logger IFF it is present, but this is somewhat clunky:
if defined?(deprecation_logger)
deprecation_logger.deprecated("some message...")
else
logger.warn("DEPRECATED: some message...")
end
Alternatively, we could ship a lightweight shim dependency that defines a Plugin#deprecation_logger if it is not already defined, which proxies calls to its deprecated method through the Plugin#logger's warn method, and ruby-based plugins that want to use the deprecation logger features can add this shim to their dependencies. We would need to be very careful to release this in a way that can be 1.0 API-stable, so as to enable major-version pinning and eliminate dependency conflicts.
Java Plugin API
If someone attempts to use the deprecation logger in their Java-based Logstash plugin, builds it, and then attempts to use the built artifact with a Logstash < 7.6, they will end up with a runtime method lookup error, likely resulting in a hard crash.
I can't see an immediate path forward on the Java Plugin API other than documenting that plugins that have been built with one version of Logstash cannot be used with prior versions of Logstash.
Perhaps this can be enforced in a "nicer" way by adding a gem dependency to the plugin packaging on logstash core ~> #{MAJOR}.#{MINOR} that it is being built with, which would ensure that the plugin couldn't install on older Logstashes than that which it was built against, but this may also end up causing significant confusion.
I think this a problem of versioning. If a plugin is for version >= "7.6" it can't run previous logstashes. We can postpone this change for LS 8.0.0 or if a plugin uses the deprecation logger it should change its requirements from a coarse dependency LS ~> 7 to a more fine grained ~> 7.6, but I don't know how to do this because the plugins only requires the gem logstash-core-plugin-api and not a "logstash gem". So we could create a new version of logstash-core-plugin-api from current 2.1.16 to the next, but many plugin express this dependency as:
s.add_runtime_dependency "logstash-core-plugin-api", ">= 1.60", "<= 2.99"
We can force the plugins that uses the deprecation logger do require the dependency as:
s.add_runtime_dependency "logstash-core-plugin-api", ">= 2.1.17", "<= 2.99"
else also if the plugins runs on LS 7.6 it can't use the deprecation logger
We cannot release minor- or patch-level updates of gems that cannot be installed on older Logstashes; this would be a breaking change, and would warrant major version bumps in the plugins, which we would not ship with Logstash until the next major (8.0.0 as of this writing).
But we could introduce a new API-stable dependency called logstash-core-deprecations-adapter, which provides Plugin#deprecation_logger if it is not already present (effectively a no-op on Logstashes >= 7.6.0). By adding this adapter as a runtime dependency to any plugin that wishes to use the deprecation logger, we can ensure that it is present (and the plugin would not need to care if it is supplied by Logstash core or by the adapter gem). I would imagine that the adapter would "fall through" to an implementation backed by the logger provided by Plugin#logger, something like:
require 'gem/requirement'
require 'gem/version'
if Gem::Requirement.new('< 7.6').satisfied_by?(Gem::Version.new(LOGSTASH_VERSION))
module DeprecationLoggerAdapter
def deprecation_logger
@deprecation_logger ||= DeprecationLoggerShim.new(self.logger)
end
class DeprecationLoggerShim
def initialize(base_logger)
@base_logger = base_logger
end
def deprecated(message, *args)
@base_logger.warn("DEPRECATED: #{message}", *args)
end
end
end
LogStash::Plugin.instance_exec { include DeprecationLoggerAdapter }
end
Something like this would solve the use-case for plugins implemented with the Ruby Plugin API.
This new dependency logstash-core-deprecations-adapter should be hosted in separate repository and provide a gem with this inside, I suppose.
For the Java part, with more work, I think we could achieve almost the same thing with a bytecode manipulation library, creating at class load time and adding a method getDeprecationLogger to co.elastic.logstash.api.Context, adding the same method to org.logstash.plugins.ContextImpl which proxy the call to a class provided by us in the new dependency. If the getDeprecationLogger method already exists then we are in LogStash >= 7.6 so bytecode manipulation is needed, else we have to provide a class that mimic that job.
Hi @yaauie I've investigated little bit to use ByteBuddy to redefine the interface co.elastic.logstash.api.Contextadding a method getDeprecationLogger to be used by plugins updated after version 7.6 running ion LS with version < 7.6. The main problem is that the HotSwap funtionality of JVM prohibit to modify the signatures of methods/classes already loaded. The possible paths are:
- redefine the class before it's loaded, so essentially mean to modify the
main()to do the job upfront all the execution. - implement ans use a Java agent that redefine the class before the classloader asks for it.
Problems with the 2 paths:
- is not viable, because require changes in code already delivered and deployed
- require to the users to append a string in the launch command of LS to include the jar with agent, jar that must be previously downloaded.
We cannot release minor- or patch-level updates of gems that cannot be installed on older Logstashes; this would be a breaking change, and would warrant major version bumps in the plugins, which we would not ship with Logstash until the next major (8.0.0 as of this writing).
We haven't guaranteed this in the past. The backwards compatibility we offer is that a pipeline configuration and logstash settings will run and produce the same results within the same major.
If minor release adds a new feature in core and new plugins are shipped that depend on this feature we only must guarantee that a bin/logstash-plugin update will not install that version, which we can ensure through logstash-core lower bound requirements.
That said, for ruby plugins it is easy for us to skip this problem by the mentioned reflection, which we also did for the dead letter queue in the ES output. The adapter idea by @yaauie is even better.
For Java, plugins are only officially supported in starting with 7.2, and I think it's ok that we ship plugins which will only work in 7.6 onwards as long as dependency resolution is used to avoid surprises at runtime.
So do we provide the gem logstash-core-deprecation-adapter that carries the Ruby adapter idea by @yaauie plus an adapter class for old LS?
Then stays to plugin developer to use the adapter and stay compatible with old LS, or use the new API but declaring the dependency on the new LS API?
Discussion wrap up:
- for the Ruby plugin we create and provide a gem (https://github.com/logstash-plugins/logstash-core-deprecation-adapter) with the ideas provided by @yaauie
- for the Java part we clearly describe in the documentation that a plugin writer that intend to use the new deprecation logger feature must bump a new a version and update the plugin dependency to LS >= 7.6 so that the plugin is not accidentally downloaded and installed by old logstashes
Hi folks, the
describe in the documentation that a plugin writer that intend to use the new deprecation logger feature must bump a new a version and update the plugin dependency to LS >= 7.6
wasn't done.
Do think we can close this or before close is better to enhance the docs?