ClassNotFoundException on Jenkins Agent shutdown caused by Otel SDK shutdown

[cyrille-leclerc]

Jenkins and plugins versions report

Environment

Paste the output here

What Operating System are you using (both controller, and any agents involved in the problem)?

2.0.0

Reproduction steps

• Enable sending pipeline logs through Otel
• Trigger a pipeline that execute on a JNLP agent
• shutdown the Jenkins Controller

Expected Results

Graceful shutdown of the Jenkins agent and of the Otel SDK on the Jenkins Agent

Actual Results

Feb 23, 2022 3:05:42 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: http://localhost:8080/login is not ready
java.net.ConnectException: Connection refused (Connection refused)
	at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:399)
	at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:242)
	at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:224)
	at java.base/java.net.Socket.connect(Socket.java:609)
	at java.base/java.net.Socket.connect(Socket.java:558)
	at java.base/sun.net.NetworkClient.doConnect(NetworkClient.java:182)
	at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:474)
	at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:569)
	at java.base/sun.net.www.http.HttpClient.<init>(HttpClient.java:242)
	at java.base/sun.net.www.http.HttpClient.New(HttpClient.java:341)
	at java.base/sun.net.www.http.HttpClient.New(HttpClient.java:362)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1253)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1187)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1081)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:1015)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1592)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1520)
	at java.base/java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:527)
	at hudson.remoting.Engine.runWebSocket(Engine.java:675)
	at hudson.remoting.Engine.run(Engine.java:496)

Feb 23, 2022 3:05:53 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: http://localhost:8080/login is not ready: 503
Feb 23, 2022 3:06:03 PM hudson.remoting.Engine lambda$new$1
SEVERE: Uncaught exception in Engine thread Thread[Thread-0,5,main]
java.lang.NoClassDefFoundError: jenkins/slaves/restarter/JnlpSlaveRestarterInstaller
	at jenkins.slaves.restarter.JnlpSlaveRestarterInstaller$FindEffectiveRestarters$1.onReconnect(JnlpSlaveRestarterInstaller.java:91)
	at hudson.remoting.EngineListenerSplitter.onReconnect(EngineListenerSplitter.java:54)
	at hudson.remoting.Engine.runWebSocket(Engine.java:687)
	at hudson.remoting.Engine.run(Engine.java:496)
Caused by: java.lang.ClassNotFoundException: jenkins.slaves.restarter.JnlpSlaveRestarterInstaller
	at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:471)
	at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:215)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:589)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
	... 4 more

Exception in thread "Thread-10" java.lang.NoClassDefFoundError: okhttp3/ConnectionPool
	at io.opentelemetry.exporter.internal.grpc.OkHttpGrpcExporter.shutdown(OkHttpGrpcExporter.java:211)
	at io.opentelemetry.exporter.otlp.trace.OtlpGrpcSpanExporter.shutdown(OtlpGrpcSpanExporter.java:75)
	at io.opentelemetry.sdk.trace.export.BatchSpanProcessor$Worker.lambda$shutdown$2(BatchSpanProcessor.java:287)
	at io.opentelemetry.sdk.common.CompletableResultCode.whenComplete(CompletableResultCode.java:126)
	at io.opentelemetry.sdk.trace.export.BatchSpanProcessor$Worker.shutdown(BatchSpanProcessor.java:284)
	at io.opentelemetry.sdk.trace.export.BatchSpanProcessor$Worker.access$200(BatchSpanProcessor.java:140)
	at io.opentelemetry.sdk.trace.export.BatchSpanProcessor.shutdown(BatchSpanProcessor.java:111)
	at io.opentelemetry.sdk.trace.TracerSharedState.shutdown(TracerSharedState.java:103)
	at io.opentelemetry.sdk.trace.SdkTracerProvider.shutdown(SdkTracerProvider.java:107)
	at io.opentelemetry.sdk.autoconfigure.AutoConfiguredOpenTelemetrySdkBuilder.lambda$build$9(AutoConfiguredOpenTelemetrySdkBuilder.java:344)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.ClassNotFoundException: okhttp3.ConnectionPool
	at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:471)
	at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:215)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:589)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
	... 11 more

Anything else?

No response

[cyrille-leclerc]

FYI Investigation in progress on https://groups.google.com/g/jenkinsci-dev/c/qotHJWEXvkg

Note that the code was cleaned up with a TODO to no longer invoke the wrong APIs (ie no longer register a JVM shutdown hook) https://github.com/jenkinsci/opentelemetry-plugin/commit/d978ac10c2ac01ef1ef93071aa160995ff5c4882

[cyrille-leclerc]

Can't reproduce