sumologic-collector-docker icon indicating copy to clipboard operation
sumologic-collector-docker copied to clipboard

Published 20 hours ago • verified publisher icon SumoLogic

java.net.SocketTimeoutException: Resource temporarily unavailable

Open axelwass opened this issue 2 years ago • 3 comments

Hi

I have installed the Sumologic collector as a Docker container.

I'm running docker on an EC2 instance on AWS. When this exception shows, I stop receiving logs from only one of the containers on the Sumologic console, this application is a Java server using spring. I can see the logs of this application using the docker logs command.

I keep getting this error:

jvm 1 | Exception in thread "onError:DockerLogManager:0000000001565CB5:'Docker-logs':44" java.lang.RuntimeException: java.net.SocketTimeoutException: Resource temporarily unavailable

Full log (until first time the error shows, then the error keeps repeating):

Running SumoLogic Collector...
wrapper  | The version of the script (3.5.49.1) doesn't match the version of this Wrapper (3.5.49). This might cause some problems
wrapper  | --> Wrapper Started as Console
wrapper  | Java Service Wrapper Standard Edition 64-bit 3.5.49
wrapper  |   Copyright (C) 1999-2022 Tanuki Software, Ltd. All Rights Reserved.
wrapper  |     http://wrapper.tanukisoftware.com
wrapper  |   Licensed to Sumo Logic Inc. for Collector
wrapper  | 
wrapper  | Launching a JVM...
jvm 1    | WrapperManager: Initializing...
jvm 1    |    . .       . .       . .    .       . .
jvm 1    | .+'|=|`+. .+'| |`+. .+'|=|`+.=|`+. .+'|=|`+.
jvm 1    | |  | `+.| |  | |  | |  | `+ | `+ | |  | |  |
jvm 1    | |  | .    |  | |  | |  |  | |  | | |  | |  |
jvm 1    | `+.|=|`+. |  | |  | |  |  | |  | | |  | |  |
jvm 1    | .    |  | |  | |  | |  |  | |  | | |  | |  |
jvm 1    | |`+. |  | |  | |  | |  |  | |  | | |  | |  |
jvm 1    | `+.|=|.+' `+.|=|.+' `+.|  |.|  |+' `+.|=|.+'
jvm 1    | Sumo Logic Collector Version 19.403-1
jvm 1    | Sumo Logic Build Hash a125300a564d
jvm 1    | current folder:/opt/SumoCollector
jvm 1    |   * See /opt/SumoCollector/./logs for more details.
jvm 1    | 2022-09-20 14:30:55,787 WrapperSimpleAppMain ERROR Appender Console cannot be located. Route ignored
jvm 1    | 2022-09-20 14:30:55,788 WrapperSimpleAppMain ERROR Appender Console cannot be located. Route ignored
jvm 1    | oslib (Utils.readProcess): Cannot run program "lsb_release": error=2, No such file or directory
jvm 1    | oslib (Utils.readProcess): Cannot run program "lsb_release": error=2, No such file or directory
jvm 1    | (oslib.linux.CommandContainsType) cmd=lsb_release -sd > null
jvm 1    |   * Connecting to https://collectors.de.sumologic.com.
jvm 1    | minor version mismatch: (7!=6) java=1.7.0.0, native=1.6.4.129
jvm 1    | org.hyperic.sigar.SigarException: minor version mismatch: (7!=6) java=1.7.0.0, native=1.6.4.129
jvm 1    |      at org.hyperic.sigar.Sigar.checkVersion(Sigar.java:148)
jvm 1    |      at org.hyperic.sigar.Sigar.<clinit>(Sigar.java:107)
jvm 1    |      at com.sumologic.collector.monitoring.SigarCollectorMetricsReporter.<init>(MetricsReporter.scala:46)
jvm 1    |      at com.sumologic.collector.monitoring.CollectorMetricsLogDumper.<init>(MetricsLogDumper.scala:49)
jvm 1    |      at com.sumologic.scala.collector.Collector$.bootstrap(Collector.scala:1176)
jvm 1    |      at com.sumologic.scala.collector.Collector$.main(Collector.scala:945)
jvm 1    |      at com.sumologic.scala.collector.Collector.main(Collector.scala)
jvm 1    |      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
jvm 1    |      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
jvm 1    |      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
jvm 1    |      at java.lang.reflect.Method.invoke(Method.java:498)
jvm 1    |      at org.tanukisoftware.wrapper.WrapperSimpleApp.run(WrapperSimpleApp.java:349)
jvm 1    |      at java.lang.Thread.run(Thread.java:750)
jvm 1    |   * Retrieved configuration from service.
jvm 1    | Exception in thread "onError:DockerLogManager:0000000001565CB5:'Docker-logs':44" java.lang.RuntimeException: java.net.SocketTimeoutException: Resource temporarily unavailable
jvm 1    |      at com.sumologic.collector.blade.docker.DockerMonitor$NativeResultCallback.onError(DockerMonitor.scala:56)
jvm 1    |      at com.github.dockerjava.core.exec.AbstrAsyncDockerCmdExec$1.onError(AbstrAsyncDockerCmdExec.java:46)
jvm 1    |      at com.github.dockerjava.jaxrs.async.AbstractCallbackNotifier.call(AbstractCallbackNotifier.java:54)
jvm 1    |      at java.lang.Thread.run(Thread.java:750)
jvm 1    | Caused by: java.net.SocketTimeoutException: Resource temporarily unavailable
jvm 1    |      at org.newsclub.net.unix.NativeUnixSocket.read(Native Method)
jvm 1    |      at org.newsclub.net.unix.AFUNIXSocketImpl$AFUNIXInputStream.read(AFUNIXSocketImpl.java:325)
jvm 1    |      at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
jvm 1    |      at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
jvm 1    |      at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280)
jvm 1    |      at org.apache.http.impl.io.ChunkedInputStream.getChunkSize(ChunkedInputStream.java:261)
jvm 1    |      at org.apache.http.impl.io.ChunkedInputStream.nextChunk(ChunkedInputStream.java:222)
jvm 1    |      at org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:183)
jvm 1    |      at org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.java:135)
jvm 1    |      at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
jvm 1    |      at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
jvm 1    |      at java.io.FilterInputStream.read(FilterInputStream.java:83)
jvm 1    |      at org.glassfish.jersey.message.internal.EntityInputStream.isEmpty(EntityInputStream.java:165)
jvm 1    |      at org.glassfish.jersey.message.internal.InboundMessageContext.hasEntity(InboundMessageContext.java:780)
jvm 1    |      at com.github.dockerjava.jaxrs.filter.LoggingFilter.filter(LoggingFilter.java:249)
jvm 1    |      at org.glassfish.jersey.client.ClientFilteringStages$ResponseFilterStage.apply(ClientFilteringStages.java:171)
jvm 1    |      at org.glassfish.jersey.client.ClientFilteringStages$ResponseFilterStage.apply(ClientFilteringStages.java:159)
jvm 1    |      at org.glassfish.jersey.process.internal.Stages.process(Stages.java:147)
jvm 1    |      at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:301)
jvm 1    |      at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$0(JerseyInvocation.java:609)
jvm 1    |      at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
jvm 1    |      at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
jvm 1    |      at org.glassfish.jersey.internal.Errors.process(Errors.java:205)
jvm 1    |      at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390)
jvm 1    |      at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:608)
jvm 1    |      at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:395)
jvm 1    |      at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:295)
jvm 1    |      at com.github.dockerjava.jaxrs.async.GETCallbackNotifier.response(GETCallbackNotifier.java:26)
jvm 1    |      at com.github.dockerjava.jaxrs.async.AbstractCallbackNotifier.call(AbstractCallbackNotifier.java:51)
jvm 1    |      ... 1 more
jvm 1    | Exception in thread "onError:DockerLogInput:0000000001565CB5:'Docker-logs':d47cd9470db84675a2e44b21159992f5710bbf6a287e6d684415eadffc657264:sumologic_docker_agent_staging:50" Exception in thread "onError:DockerStatsManager:0000000001565CB6:'Docker-stats':56" java.lang.RuntimeException: java.lang.RuntimeException: java.net.SocketTimeoutException: Resource temporarily unavailable
jvm 1    |      at com.sumologic.collector.blade.docker.DockerMonitor$NativeResultCallback.onError(DockerMonitor.scala:56)
jvm 1    |      at com.github.dockerjava.core.exec.AbstrAsyncDockerCmdExec$1.onError(AbstrAsyncDockerCmdExec.java:46)
jvm 1    |      at com.github.dockerjava.jaxrs.async.AbstractCallbackNotifier.call(AbstractCallbackNotifier.java:76)
jvm 1    |      at java.lang.Thread.run(Thread.java:750)
jvm 1    | Caused by: java.lang.RuntimeException: java.net.SocketTimeoutException: Resource temporarily unavailable
jvm 1    |      at com.sumologic.collector.blade.docker.DockerMonitor$NativeResultCallback.onError(DockerMonitor.scala:56)
jvm 1    |      at com.github.dockerjava.core.exec.AbstrAsyncDockerCmdExec$1.onError(AbstrAsyncDockerCmdExec.java:46)
jvm 1    |      at com.github.dockerjava.core.async.FrameStreamProcessor.processResponseStream(FrameStreamProcessor.java:39)
jvm 1    |      at com.github.dockerjava.jaxrs.async.AbstractCallbackNotifier.call(AbstractCallbackNotifier.java:70)
jvm 1    |      ... 1 more
jvm 1    | Caused by: java.net.SocketTimeoutException: Resource temporarily unavailable
jvm 1    |      at org.newsclub.net.unix.NativeUnixSocket.read(Native Method)
jvm 1    |      at org.newsclub.net.unix.AFUNIXSocketImpl$AFUNIXInputStream.read(AFUNIXSocketImpl.java:325)
jvm 1    |      at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
jvm 1    |      at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
jvm 1    |      at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280)
jvm 1    |      at org.apache.http.impl.io.ChunkedInputStream.getChunkSize(ChunkedInputStream.java:261)
jvm 1    |      at org.apache.http.impl.io.ChunkedInputStream.nextChunk(ChunkedInputStream.java:222)
jvm 1    |      at org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:183)
jvm 1    |      at org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.java:135)
jvm 1    |      at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
jvm 1    |      at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
jvm 1    |      at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
jvm 1    |      at java.io.FilterInputStream.read(FilterInputStream.java:133)
jvm 1    |      at org.glassfish.jersey.message.internal.EntityInputStream.read(EntityInputStream.java:79)
jvm 1    |      at com.github.dockerjava.jaxrs.util.WrappedResponseInputStream.read(WrappedResponseInputStream.java:45)
jvm 1    |      at com.github.dockerjava.core.command.FrameReader.readFrame(FrameReader.java:64)
jvm 1    |      at com.github.dockerjava.core.async.FrameStreamProcessor.processResponseStream(FrameStreamProcessor.java:35)
jvm 1    |      ... 2 more
jvm 1    | java.lang.RuntimeException: java.net.SocketTimeoutException: Resource temporarily unavailable
jvm 1    |      at com.sumologic.collector.blade.docker.DockerMonitor$NativeResultCallback.onError(DockerMonitor.scala:56)
jvm 1    |      at com.github.dockerjava.core.exec.AbstrAsyncDockerCmdExec$1.onError(AbstrAsyncDockerCmdExec.java:46)
jvm 1    |      at com.github.dockerjava.jaxrs.async.AbstractCallbackNotifier.call(AbstractCallbackNotifier.java:54)
jvm 1    |      at java.lang.Thread.run(Thread.java:750)
jvm 1    | Caused by: java.net.SocketTimeoutException: Resource temporarily unavailable
jvm 1    |      at org.newsclub.net.unix.NativeUnixSocket.read(Native Method)
jvm 1    |      at org.newsclub.net.unix.AFUNIXSocketImpl$AFUNIXInputStream.read(AFUNIXSocketImpl.java:325)
jvm 1    |      at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
jvm 1    |      at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
jvm 1    |      at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280)
jvm 1    |      at org.apache.http.impl.io.ChunkedInputStream.getChunkSize(ChunkedInputStream.java:261)
jvm 1    |      at org.apache.http.impl.io.ChunkedInputStream.nextChunk(ChunkedInputStream.java:222)
jvm 1    |      at org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:183)
jvm 1    |      at org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.java:135)
jvm 1    |      at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
jvm 1    |      at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
jvm 1    |      at java.io.FilterInputStream.read(FilterInputStream.java:83)
jvm 1    |      at org.glassfish.jersey.message.internal.EntityInputStream.isEmpty(EntityInputStream.java:165)
jvm 1    |      at org.glassfish.jersey.message.internal.InboundMessageContext.hasEntity(InboundMessageContext.java:780)
jvm 1    |      at com.github.dockerjava.jaxrs.filter.LoggingFilter.filter(LoggingFilter.java:249)
jvm 1    |      at org.glassfish.jersey.client.ClientFilteringStages$ResponseFilterStage.apply(ClientFilteringStages.java:171)
jvm 1    |      at org.glassfish.jersey.client.ClientFilteringStages$ResponseFilterStage.apply(ClientFilteringStages.java:159)
jvm 1    |      at org.glassfish.jersey.process.internal.Stages.process(Stages.java:147)
jvm 1    |      at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:301)
jvm 1    |      at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$0(JerseyInvocation.java:609)
jvm 1    |      at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
jvm 1    |      at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
jvm 1    |      at org.glassfish.jersey.internal.Errors.process(Errors.java:205)
jvm 1    |      at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390)
jvm 1    |      at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:608)
jvm 1    |      at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:395)
jvm 1    |      at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:295)
jvm 1    |      at com.github.dockerjava.jaxrs.async.GETCallbackNotifier.response(GETCallbackNotifier.java:26)
jvm 1    |      at com.github.dockerjava.jaxrs.async.AbstractCallbackNotifier.call(AbstractCallbackNotifier.java:51)
jvm 1    |      ... 1 more

axelwass avatar Sep 20 '22 15:09 axelwass

seeing the same issues with ubuntu:22.10 as there are many vulnerabilities in that ubuntu:18 base image, can we get an update on this?

that-kampe-steak avatar Nov 21 '22 19:11 that-kampe-steak

has there been any progress on the resolution of this issue? We are also experiencing problems with the same error in the logs on just one sumo logic container on one out of three swarm nodes

dsjoolbl avatar Aug 30 '23 18:08 dsjoolbl

I had a similar problem. Solved it by setting socket_timeout=0.

java.net.SocketTimeoutException: Resource temporarily unavailable
        at org.newsclub.net.unix.NativeUnixSocket.accept(Native Method)
        at org.newsclub.net.unix.AFSocketImpl.accept0(AFSocketImpl.java:290)
        at org.newsclub.net.unix.AFServerSocket.accept1(AFServerSocket.java:313)
        at org.newsclub.net.unix.AFServerSocket.accept(AFServerSocket.java:307)
        at org.newsclub.net.unix.AFUNIXServerSocket.accept(AFUNIXServerSocket.java:162)

lengdanran avatar Jul 19 '24 09:07 lengdanran