Socket timeouts

[mleegwt]

trafficstars

Great idea to make an OpenTelemetry plugin for Gradle!

When getting this to work on my local setup for development I fail to get the setup to work. My configuration is:

openTelemetryBuild {
    endpoint = "http://opentelemetry.local:4318"
    exporterMode = com.atkinsondev.opentelemetry.build.OpenTelemetryExporterMode.HTTP
}

I receive:

Failed to export spans. The request could not be executed. Full error message: timeout
java.io.InterruptedIOException: timeout
        at okhttp3.internal.connection.RealCall.timeoutExit(RealCall.kt:398)
        at okhttp3.internal.connection.RealCall.callDone(RealCall.kt:360)
        at okhttp3.internal.connection.RealCall.noMoreExchanges$okhttp(RealCall.kt:325)
        at okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:209)
        at okhttp3.internal.connection.RealCall$AsyncCall.run(RealCall.kt:517)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.net.SocketException: Socket closed
        at java.base/sun.nio.ch.NioSocketImpl.endConnect(NioSocketImpl.java:531)
        at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:615)
        at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327)
        at java.base/java.net.Socket.connect(Socket.java:633)
        at okhttp3.internal.platform.Platform.connectSocket(Platform.kt:128)
        at okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.kt:295)
        at okhttp3.internal.connection.RealConnection.connect(RealConnection.kt:207)
        at okhttp3.internal.connection.ExchangeFinder.findConnection(ExchangeFinder.kt:226)
        at okhttp3.internal.connection.ExchangeFinder.findHealthyConnection(ExchangeFinder.kt:106)
        at okhttp3.internal.connection.ExchangeFinder.find(ExchangeFinder.kt:74)
        at okhttp3.internal.connection.RealCall.initExchange$okhttp(RealCall.kt:255)
        at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.kt:32)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
        at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.kt:95)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
        at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.kt:83)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
        at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.kt:76)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
        at okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:201)
        ... 4 more

My application that I'm developing correctly connects to the HTTP setting on the URL and port above. So I'm sure the OpenTelemetry collector is set up correctly! That application is using the Java Agent default environment variables. So the endpoint setting is OTEL_EXPORTER_OTLP_ENDPOINT=http://opentelemetry.local

What could be the issue that the communication breaks down?

[craigatk]

Good question, one difference between how the Java agent handles OTEL_EXPORTER_OTLP_ENDPOINT and what this plugin does is that the Java agent adds /v1/traces to the URL when publishing traces.

Please try the URL with the traces path in it as well in the plugin configuration and see if that works for you: http://opentelemetry.local:4318/v1/traces

If that fixes the problem, please let me know and I'll update the docs to include a note about that for the HTTP exporter.

[mleegwt]

Alas, unfortunately the same stacktrace occurs when using /v1/traces as well.

[craigatk]

Bummer! The next thing I'd try is using the gRCP exporter instead of the HTTP one and see if that makes any difference.

[mleegwt]

I've tried that as well. But it didn't work. A colleague will assist me some time in the future. We might be able to progress on this issue. I'll let you know what happens.

[ritesh-singh]

Spinned local instance of JegerUI

docker run -d --name jaeger \
  -e COLLECTOR_ZIPKIN_HTTP_PORT=9411 \
  -p 5775:5775/udp \
  -p 6831:6831/udp \
  -p 6832:6832/udp \
  -p 5778:5778 \
  -p 16686:16686 \
  -p 14268:14268 \
  -p 9411:9411 \
  jaegertracing/all-in-one:1.6

and configuration as

openTelemetryBuild {
  endpoint = "http://localhost:16686/"
  exporterMode = OpenTelemetryExporterMode.HTTP
}

Build, succeeds without any error - but i don't see any traces on exporter.

[craigatk]

@ritesh-singh good question, with a couple tweaks to that config I was able to get traces going to Jaegar locally.

I wrote up the instructions in a new section in the project readme: https://github.com/craigatk/opentelemetry-gradle-plugin?tab=readme-ov-file#jaeger

[ritesh-singh]

Thanks @craigatk - 🙌 I can see the traces.

[mleegwt]

Sorry for the long delay. Your writeup works. Nice! Unfortunately after enabling the plugin I also have the following stacktrace:

Unexpected exception thrown.
org.gradle.internal.remote.internal.MessageIOException: Could not write '/127.0.0.1:61099'.
        at org.gradle.internal.remote.internal.inet.SocketConnection.flush(SocketConnection.java:140)
        at org.gradle.internal.remote.internal.hub.MessageHub$ConnectionDispatch.run(MessageHub.java:333)
        at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
        at org.gradle.internal.concurrent.AbstractManagedExecutor$1.run(AbstractManagedExecutor.java:47)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.io.IOException: Connection reset by peer
        at java.base/sun.nio.ch.SocketDispatcher.write0(Native Method)
        at java.base/sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:54)
        at java.base/sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:132)
        at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:76)
        at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:53)
        at java.base/sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:532)
        at org.gradle.internal.remote.internal.inet.SocketConnection$SocketOutputStream.writeWithNonBlockingRetry(SocketConnection.java:279)
        at org.gradle.internal.remote.internal.inet.SocketConnection$SocketOutputStream.writeBufferToChannel(SocketConnection.java:267)
        at org.gradle.internal.remote.internal.inet.SocketConnection$SocketOutputStream.flush(SocketConnection.java:261)
        at org.gradle.internal.remote.internal.inet.SocketConnection.flush(SocketConnection.java:138)
        ... 6 more

But this is a new issue. So I'll close this one and open a new one.