digdag
digdag copied to clipboard
[feature request] Lifecycle management plugin interface OR Global resource management plugin interface
Hi all, I have a feature request: the lifecycle management plugin interface OR the global resource management plugin interface.
I wrote a plugin digdag-operator-pg_lock that can get locks by PostgreSQL(if you want to know detailed use case, see the plugin README). However I cannot control the connection pool effectively. This is because the current digdag plugin interface does not manage the lifecycle nor the global resource. I want to close the pool when the plugin cache is expired or to define the global connection pool.
Currently the plugin interface has configureBinder
, but, even if I define a class as a singleton object in this method, the class does not become singleton.
https://github.com/treasure-data/digdag/blob/aaefa00879e2a53bce21bd5df440fa6d0a02a461/digdag-spi/src/main/java/io/digdag/spi/Plugin.java#L9-L10
Because the plugin cache is expired in 10 minutes and re-create the plugin with a new child injector (and the injector is not referenced anywhere after creating the providerClass
instance).
https://github.com/treasure-data/digdag/blob/aaefa00879e2a53bce21bd5df440fa6d0a02a461/digdag-core/src/main/java/io/digdag/core/plugin/DynamicPluginLoader.java#L47
https://github.com/treasure-data/digdag/blob/aaefa00879e2a53bce21bd5df440fa6d0a02a461/digdag-core/src/main/java/io/digdag/core/plugin/DynamicPluginLoader.java#L67
https://github.com/treasure-data/digdag/blob/aaefa00879e2a53bce21bd5df440fa6d0a02a461/digdag-core/src/main/java/io/digdag/core/plugin/PluginSet.java#L26-L45
In addition, Guice does not manage the lifecycle. https://github.com/google/guice/issues/62 https://github.com/google/guice/issues/1069
So I'm in trouble because I can't implement a connection pool well. For the current digdag-operator-pg_lock implementation, the connection pool is closed with the finalize
method, but it's not very good, I think.
That's why I want the feature. If you have any other good advice, please let me know...