Reconnect fails

[tmichels-brightplan]

Running on Heroku seems to throw error when trying to stop cometd

2018-10-02T03:15:42.243201+00:00 app[worker.1]: <<<<
2018-10-02T03:15:42.243161+00:00 app[worker.1]: {advice={reconnect=handshake, interval=0}, channel=/meta/connect, id=297, error=403::Unknown client, successful=false}
2018-10-02T03:15:42.244849+00:00 app[worker.1]: >>>>
2018-10-02T03:15:42.245023+00:00 app[worker.1]: [2018-10-02 03:15:42.244] Failure:[/meta/disconnect]
2018-10-02T03:15:42.245060+00:00 app[worker.1]: {failure={exception=java.nio.channels.AsynchronousCloseException, message={clientId=391c3uwfrupcxgf1uaexxljm6fow, channel=/meta/disconnect, id=298}, connectionType=long-polling}, channel=/meta/disconnect, id=298, successful=false}
2018-10-02T03:15:42.245063+00:00 app[worker.1]: <<<<
2018-10-02T03:15:42.245086+00:00 app[worker.1]: >>>>
2018-10-02T03:15:42.245199+00:00 app[worker.1]: [2018-10-02 03:15:42.245] Failure:[/meta/disconnect]
2018-10-02T03:15:42.245201+00:00 app[worker.1]: {failure={exception=java.nio.channels.AsynchronousCloseException, message={clientId=391c3uwfrupcxgf1uaexxljm6fow, channel=/meta/disconnect, id=298}, connectionType=long-polling}, channel=/meta/disconnect, id=298, successful=false}
2018-10-02T03:15:42.245202+00:00 app[worker.1]: <<<<
2018-10-02T03:15:57.247250+00:00 app[worker.1]: [HttpClient@411721666-48] ERROR c.s.e.c.EmpConnector - Unable to stop HTTP transport[https://cs43.salesforce.com/cometd/37.0]
2018-10-02T03:15:57.247263+00:00 app[worker.1]: java.lang.InterruptedException: null
2018-10-02T03:15:57.247265+00:00 app[worker.1]: at java.lang.Object.wait(Native Method)
2018-10-02T03:15:57.247266+00:00 app[worker.1]: at java.lang.Thread.join(Thread.java:1260)
2018-10-02T03:15:57.247269+00:00 app[worker.1]: at org.eclipse.jetty.util.thread.QueuedThreadPool.doStop(QueuedThreadPool.java:154)
2018-10-02T03:15:57.247271+00:00 app[worker.1]: at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:89)
2018-10-02T03:15:57.247272+00:00 app[worker.1]: at org.eclipse.jetty.util.component.ContainerLifeCycle.stop(ContainerLifeCycle.java:143)
2018-10-02T03:15:57.247274+00:00 app[worker.1]: at org.eclipse.jetty.util.component.ContainerLifeCycle.doStop(ContainerLifeCycle.java:162)
2018-10-02T03:15:57.247276+00:00 app[worker.1]: at org.eclipse.jetty.client.HttpClient.doStop(HttpClient.java:252)
2018-10-02T03:15:57.247278+00:00 app[worker.1]: at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:89)
2018-10-02T03:15:57.247280+00:00 app[worker.1]: at com.salesforce.emp.connector.EmpConnector.stop(EmpConnector.java:154)
2018-10-02T03:15:57.247282+00:00 app[worker.1]: at com.salesforce.emp.connector.EmpConnector$AuthFailureListener.onMessage(EmpConnector.java:334)
2018-10-02T03:15:57.339835+00:00 app[worker.1]: >>>>
2018-10-02T03:15:57.339956+00:00 app[worker.1]: [2018-10-02 03:15:57.339] Success:[/meta/handshake]
2018-10-02T03:15:57.339991+00:00 app[worker.1]: {ext={replay=true, payload.format=true}, minimumVersion=1.0, clientId=3bd0a4rek46fkg14j04vgo0dbb0, supportedConnectionTypes=[Ljava.lang.Object;@5ebb9ec0, channel=/meta/handshake, id=299, version=1.0, successful=true}

[choidea]

I met this issue too, and can't find the answer.

[hnguyen0971]

I see this issue also.

[vcarnogu]

Same problem here. Running on my local machine. From the logs you can see it will eventually re-connect but timeouts are very long. Sending message to /meta/disconnect fail as well as stopping of HttpClient.

[avromf]

I've encountered this issue as well. Specifically, the issue is in the stop method which fails due to an InterruptedException, reconnect works fine.

[lettl]

I believe the reason this fails is due to the way the org.eclipse.jetty.client.HttpClient is being used in EmpConnector.connect() and EmpConnector.stop(). The EmpConnector.AuthFailureListener calls EmpConnector.stop() during onMessage() if the proper conditions are met. This makes sense, but if one looks at the stop method, line 157 indicates that stop() is also being called on the org.eclipse.jetty.client.HttpClient which is essentially a shutdown operation. I don't believe this is desirable based on the documentation for the client which indicates that org.eclipse.jetty.client.HttpClient.stop() should be called just prior to application shutdown.

Additionally, the EmpConnector.connect() method should only call org.eclipse.jetty.client.HttpClient.start() if it is not started or in the process of starting, so adding a conditional block to check the startup status before calling org.eclipse.jetty.client.HttpClient.start() is suggested

Finally, the EmpConnector should provide a method that does call org.eclipse.jetty.client.HttpClient.stop()

[mohitj13]

Encountering the same issue.

[madzskill]

Encountered the same issue here, when running in local. Any solution or temporary fix to this error? Thanks.

[madzskill]

@pbn-sfdc @sfdc-hhildebrand Is this issue in anyway, related to this: https://github.com/forcedotcom/EMP-Connector/issues/15

[One-Boat]

Encountering the same issue. `Failure:[/meta/connect] {advice={reconnect=handshake, interval=0}, channel=/meta/connect, id=105, error=403::Unknown client, successful=false} <<<< 2019-07-11 14:59:18.668 INFO 28657 --- [ient@a1f72f5-50] c.y.c.d.empConnector.EmpConnector : Disconnecting Bayeux Client in EmpConnector

[2019-07-11 14:59:18.673] Failure:[/meta/disconnect] {failure={exception=java.nio.channels.AsynchronousCloseException, message={clientId=425wcjiq616k9fsjwviwtay3d4, channel=/meta/disconnect, id=106}, connectionType=long-polling}, channel=/meta/disconnect, id=106, successful=false} <<<< 2019-07-11 14:59:48.688 ERROR 28657 --- [ient@a1f72f5-50] c.y.c.d.empConnector.EmpConnector : Unable to stop HTTP transport[https://test.salesforce.com/cometd/43.0]

java.lang.RuntimeException: java.lang.InterruptedException at org.eclipse.jetty.util.component.ContainerLifeCycle.remove(ContainerLifeCycle.java:627) ~[jetty-util-9.4.19.v20190610.jar!/:9.4.19.v20190610] at org.eclipse.jetty.util.component.ContainerLifeCycle.removeBean(ContainerLifeCycle.java:597) ~[jetty-util-9.4.19.v20190610.jar!/:9.4.19.v20190610] at org.eclipse.jetty.io.SelectorManager.doStop(SelectorManager.java:289) ~[jetty-io-9.4.19.v20190610.jar!/:9.4.19.v20190610] at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:89) ~[jetty-util-9.4.19.v20190610.jar!/:9.4.19.v20190610] at org.eclipse.jetty.util.component.ContainerLifeCycle.stop(ContainerLifeCycle.java:178) ~[jetty-util-9.4.19.v20190610.jar!/:9.4.19.v20190610] at org.eclipse.jetty.util.component.ContainerLifeCycle.doStop(ContainerLifeCycle.java:199) ~[jetty-util-9.4.19.v20190610.jar!/:9.4.19.v20190610] at org.eclipse.jetty.client.AbstractConnectorHttpClientTransport.doStop(AbstractConnectorHttpClientTransport.java:70) ~[jetty-client-9.4.19.v20190610.jar!/:9.4.19.v20190610] at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:89) ~[jetty-util-9.4.19.v20190610.jar!/:9.4.19.v20190610] at org.eclipse.jetty.util.component.ContainerLifeCycle.stop(ContainerLifeCycle.java:178) ~[jetty-util-9.4.19.v20190610.jar!/:9.4.19.v20190610] at org.eclipse.jetty.util.component.ContainerLifeCycle.doStop(ContainerLifeCycle.java:199) ~[jetty-util-9.4.19.v20190610.jar!/:9.4.19.v20190610] at org.eclipse.jetty.client.HttpClient.doStop(HttpClient.java:266) ~[jetty-client-9.4.19.v20190610.jar!/:9.4.19.v20190610] at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:89) ~[jetty-util-9.4.19.v20190610.jar!/:9.4.19.v20190610] at com.yuexiu.crm.datasync.empConnector.EmpConnector.stop(EmpConnector.java:146) ~[classes!/:0.0.1-SNAPSHOT] at com.yuexiu.crm.datasync.empConnector.EmpConnector$AuthFailureListener.onMessage(EmpConnector.java:342) [classes!/:0.0.1-SNAPSHOT] at org.cometd.common.AbstractClientSession$AbstractSessionChannel.notifyOnMessage(AbstractClientSession.java:597) [cometd-java-common-3.1.4.jar!/:na] at org.cometd.common.AbstractClientSession$AbstractSessionChannel.notifyMessageListeners(AbstractClientSession.java:582) [cometd-java-common-3.1.4.jar!/:na] at org.cometd.common.AbstractClientSession.notifyListeners(AbstractClientSession.java:294) [cometd-java-common-3.1.4.jar!/:na] at org.cometd.common.AbstractClientSession.receive(AbstractClientSession.java:260) [cometd-java-common-3.1.4.jar!/:na] at org.cometd.client.BayeuxClient.failConnect(BayeuxClient.java:776) [cometd-java-client-3.1.4.jar!/:na] at org.cometd.client.BayeuxClient.processConnect(BayeuxClient.java:762) [cometd-java-client-3.1.4.jar!/:na] at org.cometd.client.BayeuxClient.processMessages(BayeuxClient.java:610) [cometd-java-client-3.1.4.jar!/:na] at org.cometd.client.BayeuxClient.access$3100(BayeuxClient.java:100) [cometd-java-client-3.1.4.jar!/:na] at org.cometd.client.BayeuxClient$MessageTransportListener.onMessages(BayeuxClient.java:1190) [cometd-java-client-3.1.4.jar!/:na] at org.cometd.client.transport.LongPollingTransport$2.onComplete(LongPollingTransport.java:236) [cometd-java-client-3.1.4.jar!/:na] at org.eclipse.jetty.client.ResponseNotifier.notifyComplete(ResponseNotifier.java:196) [jetty-client-9.4.19.v20190610.jar!/:9.4.19.v20190610] at org.eclipse.jetty.client.ResponseNotifier.notifyComplete(ResponseNotifier.java:188) [jetty-client-9.4.19.v20190610.jar!/:9.4.19.v20190610] at org.eclipse.jetty.client.HttpReceiver.terminateResponse(HttpReceiver.java:441) [jetty-client-9.4.19.v20190610.jar!/:9.4.19.v20190610] at org.eclipse.jetty.client.HttpReceiver.responseSuccess(HttpReceiver.java:387) [jetty-client-9.4.19.v20190610.jar!/:9.4.19.v20190610] at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.messageComplete(HttpReceiverOverHTTP.java:316) [jetty-client-9.4.19.v20190610.jar!/:9.4.19.v20190610] at org.eclipse.jetty.http.HttpParser.handleContentMessage(HttpParser.java:600) [jetty-http-9.4.19.v20190610.jar!/:9.4.19.v20190610] at org.eclipse.jetty.http.HttpParser.parseContent(HttpParser.java:1670) [jetty-http-9.4.19.v20190610.jar!/:9.4.19.v20190610] at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:1518) [jetty-http-9.4.19.v20190610.jar!/:9.4.19.v20190610] at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.parse(HttpReceiverOverHTTP.java:172) [jetty-client-9.4.19.v20190610.jar!/:9.4.19.v20190610] at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.process(HttpReceiverOverHTTP.java:135) [jetty-client-9.4.19.v20190610.jar!/:9.4.19.v20190610] at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.receive(HttpReceiverOverHTTP.java:73) [jetty-client-9.4.19.v20190610.jar!/:9.4.19.v20190610] at org.eclipse.jetty.client.http.HttpChannelOverHTTP.receive(HttpChannelOverHTTP.java:133) [jetty-client-9.4.19.v20190610.jar!/:9.4.19.v20190610] at org.eclipse.jetty.client.http.HttpConnectionOverHTTP.onFillable(HttpConnectionOverHTTP.java:155) [jetty-client-9.4.19.v20190610.jar!/:9.4.19.v20190610] at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305) [jetty-io-9.4.19.v20190610.jar!/:9.4.19.v20190610] at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) [jetty-io-9.4.19.v20190610.jar!/:9.4.19.v20190610] at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:427) [jetty-io-9.4.19.v20190610.jar!/:9.4.19.v20190610] at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:321) [jetty-io-9.4.19.v20190610.jar!/:9.4.19.v20190610] at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:159) [jetty-io-9.4.19.v20190610.jar!/:9.4.19.v20190610] at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) [jetty-io-9.4.19.v20190610.jar!/:9.4.19.v20190610] at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117) [jetty-io-9.4.19.v20190610.jar!/:9.4.19.v20190610] at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333) [jetty-util-9.4.19.v20190610.jar!/:9.4.19.v20190610] at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310) [jetty-util-9.4.19.v20190610.jar!/:9.4.19.v20190610] at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) [jetty-util-9.4.19.v20190610.jar!/:9.4.19.v20190610] at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126) [jetty-util-9.4.19.v20190610.jar!/:9.4.19.v20190610] at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366) [jetty-util-9.4.19.v20190610.jar!/:9.4.19.v20190610] at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:781) [jetty-util-9.4.19.v20190610.jar!/:9.4.19.v20190610] at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:917) [jetty-util-9.4.19.v20190610.jar!/:9.4.19.v20190610] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_144]`

[DeepaChakkunny]

Is there any fix for this issue. It is always disconnecting after 3-4 hours.

[mohitj13]

PR https://github.com/forcedotcom/EMP-Connector/pull/62

[madzskill]

@mohitj13 Did this fix the issue? Thanks.

[mohitj13]

Hi @madzskill this fix is in regards to comment https://github.com/forcedotcom/EMP-Connector/issues/40#issuecomment-444305378. This does not fix the disconnection but it tries to successfully reconnects when gets a disconnect message, as it does not stop the httpClient and just reconnect the Bayeux client only.

[madzskill]

I believe the reason this fails is due to the way the org.eclipse.jetty.client.HttpClient is being used in EmpConnector.connect() and EmpConnector.stop(). The EmpConnector.AuthFailureListener calls EmpConnector.stop() during onMessage() if the proper conditions are met. This makes sense, but if one looks at the stop method, line 157 indicates that stop() is also being called on the org.eclipse.jetty.client.HttpClient which is essentially a shutdown operation. I don't believe this is desirable based on the documentation for the client which indicates that org.eclipse.jetty.client.HttpClient.stop() should be called just prior to application shutdown.

Additionally, the EmpConnector.connect() method should only call org.eclipse.jetty.client.HttpClient.start() if it is not started or in the process of starting, so adding a conditional block to check the startup status before calling org.eclipse.jetty.client.HttpClient.start() is suggested

Finally, the EmpConnector should provide a method that does call org.eclipse.jetty.client.HttpClient.stop()

@lettl Did you find solution to this? Maybe, you can pull request it. Thanks.

[lettl]

@madzskill - I implemented my own solution just because it was going to be necessary to change the public contract for EMP-connector in order to fix properly. I looked at the PR that @mohitj13 created and it looks good and should also resolve the reconnect failures and problems with shutting down the http client. With my implementation (very similar to what @mohitj13 did) I am able to successfully reconnect after receiving the auth failure message which happens about every 3 hours. Once @mohitj13 change merges, that should resolve the reconnect failure issue.

[kubacech]

Encountering the same issue too and it looks like PR#62 from @mohitj13 solves this issue

[madzskill]

@lettl Thanks for the info, we're good then. Hope the original author or someone with authority can approve and merge mohitj13's changes, since a lot of us here are encountering this reconnect failure issue..

[chandu1988]

Is there any fix for this issue. It is always disconnecting after 3-4 hours.

[chandu1988]

i am getting this error so please provide the solution any one knows 06-12-2019 12:18:12.287 [main] INFO c.s.e.c.e.EmpconnectorApplication.logStartupProfileInfo - No active profile set, falling back to default profiles: default 06-12-2019 12:18:12.971 [main] INFO c.s.e.c.e.EmpconnectorApplication.logStarted - Started EmpconnectorApplication in 1.101 seconds (JVM running for 1.527) 06-12-2019 12:18:12.997 [main] INFO org.eclipse.jetty.util.log.initialized - Logging initialized @1551ms to org.eclipse.jetty.util.log.Slf4jLog 06-12-2019 12:18:13.303 [main] INFO c.s.e.c.LoginHelper$LoginResponseParser.login - client start: .... 06-12-2019 12:18:14.470 [main] INFO c.s.e.c.e.EmpconnectorApplication.subscriptionEventsSalesForce - token:: 06-12-2019 12:18:14.471 [main] INFO c.s.emp.connector.EmpConnector.connect - EmpConnector connecting 06-12-2019 12:18:15.102 [main] INFO c.s.emp.connector.EmpConnector.subscribe - topic :/event/OutBound_PE__e 06-12-2019 12:18:15.102 [main] INFO c.s.emp.connector.EmpConnector.subscribe - /event/OutBound_PE__eHI 06-12-2019 12:18:15.659 [main] INFO c.s.e.c.e.EmpconnectorApplication.subscriptionEventsSalesForce - Subscribed: Subscription [/event/OutBound_PE__e:-1] 06-12-2019 12:18:15.660 [main] INFO c.s.e.c.e.EmpconnectorApplication.main - ---------start---------- 06-12-2019 18:23:37.327 [HttpClient@42eca56e-26] INFO c.s.emp.connector.EmpConnector.stop - Disconnecting Bayeux Client in EmpConnector 06-12-2019 18:24:07.335 [HttpClient@42eca56e-26] ERROR c.s.emp.connector.EmpConnector.stop - Unable to stop HTTP transport[https://test.salesforce.com] java.lang.RuntimeException: java.lang.InterruptedException at org.eclipse.jetty.util.component.ContainerLifeCycle.remove(ContainerLifeCycle.java:598) at org.eclipse.jetty.util.component.ContainerLifeCycle.removeBean(ContainerLifeCycle.java:568) at org.eclipse.jetty.io.SelectorManager.doStop(SelectorManager.java:289) at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:89) at org.eclipse.jetty.util.component.ContainerLifeCycle.stop(ContainerLifeCycle.java:149) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStop(ContainerLifeCycle.java:170) at org.eclipse.jetty.client.AbstractConnectorHttpClientTransport.doStop(AbstractConnectorHttpClientTransport.java:70) at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:89) at org.eclipse.jetty.util.component.ContainerLifeCycle.stop(ContainerLifeCycle.java:149) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStop(ContainerLifeCycle.java:170) at org.eclipse.jetty.client.HttpClient.doStop(HttpClient.java:266) at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:89) at com.salesforce.emp.connector.EmpConnector.stop(EmpConnector.java:155) at com.salesforce.emp.connector.EmpConnector$AuthFailureListener.onMessage(EmpConnector.java:344) at org.cometd.common.AbstractClientSession$AbstractSessionChannel.notifyOnMessage(AbstractClientSession.java:597) at org.cometd.common.AbstractClientSession$AbstractSessionChannel.notifyMessageListeners(AbstractClientSession.java:582) at org.cometd.common.AbstractClientSession.notifyListeners(AbstractClientSession.java:294) at org.cometd.common.AbstractClientSession.receive(AbstractClientSession.java:260) at org.cometd.client.BayeuxClient.failConnect(BayeuxClient.java:776) at org.cometd.client.BayeuxClient.processConnect(BayeuxClient.java:762) at org.cometd.client.BayeuxClient.processMessages(BayeuxClient.java:610) at org.cometd.client.BayeuxClient.access$3100(BayeuxClient.java:100) at org.cometd.client.BayeuxClient$MessageTransportListener.onMessages(BayeuxClient.java:1190) at org.cometd.client.transport.LongPollingTransport$2.onComplete(LongPollingTransport.java:236) at org.eclipse.jetty.client.ResponseNotifier.notifyComplete(ResponseNotifier.java:196) at org.eclipse.jetty.client.ResponseNotifier.notifyComplete(ResponseNotifier.java:188) at org.eclipse.jetty.client.HttpReceiver.terminateResponse(HttpReceiver.java:470) at org.eclipse.jetty.client.HttpReceiver.responseSuccess(HttpReceiver.java:416) at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.messageComplete(HttpReceiverOverHTTP.java:316) at org.eclipse.jetty.http.HttpParser.handleContentMessage(HttpParser.java:599) at org.eclipse.jetty.http.HttpParser.parseContent(HttpParser.java:1669) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:1517) at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.parse(HttpReceiverOverHTTP.java:172) at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.process(HttpReceiverOverHTTP.java:135) at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.receive(HttpReceiverOverHTTP.java:73) at org.eclipse.jetty.client.http.HttpChannelOverHTTP.receive(HttpChannelOverHTTP.java:133) at org.eclipse.jetty.client.http.HttpConnectionOverHTTP.onFillable(HttpConnectionOverHTTP.java:155) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:427) at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:321) at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:159) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.InterruptedException: null at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1260) at org.eclipse.jetty.util.thread.QueuedThreadPool.doStop(QueuedThreadPool.java:188) at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:89) at org.eclipse.jetty.util.component.ContainerLifeCycle.stop(ContainerLifeCycle.java:149) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStop(ContainerLifeCycle.java:170) at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:89) at org.eclipse.jetty.util.component.ContainerLifeCycle.stop(ContainerLifeCycle.java:149) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStop(ContainerLifeCycle.java:170) at org.eclipse.jetty.io.ManagedSelector.doStop(ManagedSelector.java:150) at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:89) at org.eclipse.jetty.util.component.ContainerLifeCycle.stop(ContainerLifeCycle.java:149) at org.eclipse.jetty.util.component.ContainerLifeCycle.remove(ContainerLifeCycle.java:590) ... 51 common frames omitted 06-12-2019 18:24:07.336 [HttpClient@42eca56e-26] INFO c.s.emp.connector.EmpConnector.connect - EmpConnector connecting 06-12-2019 18:24:07.337 [HttpClient@42eca56e-26] WARN o.e.j.util.thread.QueuedThreadPool.execute - QueuedThreadPool[HttpClient@42eca56e]@569cfc36{FAILED,8<=8<=200,i=0,q=9}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$169/366004251@7204defe] rejected org.eclipse.jetty.io.ManagedSelector$$Lambda$174/733957003@37065eda 06-12-2019 18:24:07.338 [HttpClient@42eca56e-26] ERROR c.s.emp.connector.EmpConnector.connect - Unable to start HTTP transport[https://test.salesforce.com] java.util.concurrent.RejectedExecutionException: org.eclipse.jetty.io.ManagedSelector$$Lambda$174/733957003@37065eda at org.eclipse.jetty.util.thread.QueuedThreadPool.execute(QueuedThreadPool.java:440) at org.eclipse.jetty.io.SelectorManager.execute(SelectorManager.java:163) at org.eclipse.jetty.io.ManagedSelector.doStart(ManagedSelector.java:116) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:108) at org.eclipse.jetty.io.SelectorManager.doStart(SelectorManager.java:262) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:108) at org.eclipse.jetty.client.AbstractConnectorHttpClientTransport.doStart(AbstractConnectorHttpClientTransport.java:64) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:108) at org.eclipse.jetty.client.HttpClient.doStart(HttpClient.java:244) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) at com.salesforce.emp.connector.EmpConnector.connect(EmpConnector.java:263) at com.salesforce.emp.connector.EmpConnector.reconnect(EmpConnector.java:325) at com.salesforce.emp.connector.EmpConnector.access$1100(EmpConnector.java:31) at com.salesforce.emp.connector.EmpConnector$AuthFailureListener.onMessage(EmpConnector.java:345) at org.cometd.common.AbstractClientSession$AbstractSessionChannel.notifyOnMessage(AbstractClientSession.java:597) at org.cometd.common.AbstractClientSession$AbstractSessionChannel.notifyMessageListeners(AbstractClientSession.java:582) at org.cometd.common.AbstractClientSession.notifyListeners(AbstractClientSession.java:294) at org.cometd.common.AbstractClientSession.receive(AbstractClientSession.java:260) at org.cometd.client.BayeuxClient.failConnect(BayeuxClient.java:776) at org.cometd.client.BayeuxClient.processConnect(BayeuxClient.java:762) at org.cometd.client.BayeuxClient.processMessages(BayeuxClient.java:610) at org.cometd.client.BayeuxClient.access$3100(BayeuxClient.java:100) at org.cometd.client.BayeuxClient$MessageTransportListener.onMessages(BayeuxClient.java:1190) at org.cometd.client.transport.LongPollingTransport$2.onComplete(LongPollingTransport.java:236) at org.eclipse.jetty.client.ResponseNotifier.notifyComplete(ResponseNotifier.java:196) at org.eclipse.jetty.client.ResponseNotifier.notifyComplete(ResponseNotifier.java:188) at org.eclipse.jetty.client.HttpReceiver.terminateResponse(HttpReceiver.java:470) at org.eclipse.jetty.client.HttpReceiver.responseSuccess(HttpReceiver.java:416) at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.messageComplete(HttpReceiverOverHTTP.java:316) at org.eclipse.jetty.http.HttpParser.handleContentMessage(HttpParser.java:599) at org.eclipse.jetty.http.HttpParser.parseContent(HttpParser.java:1669) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:1517) at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.parse(HttpReceiverOverHTTP.java:172) at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.process(HttpReceiverOverHTTP.java:135) at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.receive(HttpReceiverOverHTTP.java:73) at org.eclipse.jetty.client.http.HttpChannelOverHTTP.receive(HttpChannelOverHTTP.java:133) at org.eclipse.jetty.client.http.HttpConnectionOverHTTP.onFillable(HttpConnectionOverHTTP.java:155) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:427) at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:321) at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:159) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683) at java.lang.Thread.run(Thread.java:748) 06-12-2019 18:24:07.338 [HttpClient@42eca56e-26] INFO c.s.emp.connector.EmpConnector.onMessage - reconect........

[jordan-mathews]

Encountering the same issue. Has anyone gotten these errors recently? Any suggestions cause and possible fix?