docker-apim icon indicating copy to clipboard operation
docker-apim copied to clipboard

Published 20 hours ago • verified publisher icon wso2

SSL-related exceptions with connection to ssl://am-analytics:7712

Open jeremy303 opened this issue 6 years ago • 8 comments

Description:

Receiving the following exceptions in api-manager, running the current master branch out-of-the-box:

api-manager_1   | [2018-03-28 21:00:54,702] ERROR - DataEndpointConnectionWorker Error while trying to connect to the endpoint. Cannot borrow client for ssl://am-analytics:7712
api-manager_1   | org.wso2.carbon.databridge.agent.exception.DataEndpointAuthenticationException: Cannot borrow client for ssl://am-analytics:7712
api-manager_1   | 	at org.wso2.carbon.databridge.agent.endpoint.DataEndpointConnectionWorker.connect(DataEndpointConnectionWorker.java:99)
api-manager_1   | 	at org.wso2.carbon.databridge.agent.endpoint.DataEndpointConnectionWorker.run(DataEndpointConnectionWorker.java:42)
api-manager_1   | 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
api-manager_1   | 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
api-manager_1   | 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
api-manager_1   | 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
api-manager_1   | 	at java.lang.Thread.run(Thread.java:748)
api-manager_1   | Caused by: org.wso2.carbon.databridge.agent.exception.DataEndpointAuthenticationException: Thrift exception
api-manager_1   | 	at org.wso2.carbon.databridge.agent.endpoint.thrift.ThriftDataEndpoint.login(ThriftDataEndpoint.java:49)
api-manager_1   | 	at org.wso2.carbon.databridge.agent.endpoint.DataEndpointConnectionWorker.connect(DataEndpointConnectionWorker.java:93)
api-manager_1   | 	... 6 more
api-manager_1   | Caused by: org.apache.thrift.transport.TTransportException: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: signature check failed
api-manager_1   | 	at org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:161)
api-manager_1   | 	at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:65)
api-manager_1   | 	at org.wso2.carbon.databridge.commons.thrift.service.secure.ThriftSecureEventTransmissionService$Client.send_connect(ThriftSecureEventTransmissionService.java:104)
api-manager_1   | 	at org.wso2.carbon.databridge.commons.thrift.service.secure.ThriftSecureEventTransmissionService$Client.connect(ThriftSecureEventTransmissionService.java:95)
api-manager_1   | 	at org.wso2.carbon.databridge.agent.endpoint.thrift.ThriftDataEndpoint.login(ThriftDataEndpoint.java:45)
api-manager_1   | 	... 7 more
api-manager_1   | Caused by: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: signature check failed
api-manager_1   | 	at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
api-manager_1   | 	at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1959)
api-manager_1   | 	at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:328)
api-manager_1   | 	at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:322)
api-manager_1   | 	at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1614)
api-manager_1   | 	at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216)
api-manager_1   | 	at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052)
api-manager_1   | 	at sun.security.ssl.Handshaker.process_record(Handshaker.java:987)
api-manager_1   | 	at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1072)
api-manager_1   | 	at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1385)
api-manager_1   | 	at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:757)
api-manager_1   | 	at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:123)
api-manager_1   | 	at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
api-manager_1   | 	at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
api-manager_1   | 	at org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:159)
api-manager_1   | 	... 11 more
api-manager_1   | Caused by: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: signature check failed
api-manager_1   | 	at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:362)
api-manager_1   | 	at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:270)
api-manager_1   | 	at sun.security.validator.Validator.validate(Validator.java:260)
api-manager_1   | 	at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324)
api-manager_1   | 	at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:229)
api-manager_1   | 	at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:124)
api-manager_1   | 	at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1596)
api-manager_1   | 	... 21 more
api-manager_1   | Caused by: java.security.cert.CertPathValidatorException: signature check failed
api-manager_1   | 	at sun.security.provider.certpath.PKIXMasterCertPathValidator.validate(PKIXMasterCertPathValidator.java:135)
api-manager_1   | 	at sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:223)
api-manager_1   | 	at sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:140)
api-manager_1   | 	at sun.security.provider.certpath.PKIXCertPathValidator.engineValidate(PKIXCertPathValidator.java:79)
api-manager_1   | 	at java.security.cert.CertPathValidator.validate(CertPathValidator.java:292)
api-manager_1   | 	at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:357)
api-manager_1   | 	... 27 more
api-manager_1   | Caused by: java.security.SignatureException: Signature length not correct: got 256 but was expecting 128
api-manager_1   | 	at sun.security.rsa.RSASignature.engineVerify(RSASignature.java:189)
api-manager_1   | 	at java.security.Signature$Delegate.engineVerify(Signature.java:1223)
api-manager_1   | 	at java.security.Signature.verify(Signature.java:656)
api-manager_1   | 	at sun.security.x509.X509CertImpl.verify(X509CertImpl.java:444)
api-manager_1   | 	at sun.security.provider.certpath.BasicChecker.verifySignature(BasicChecker.java:166)
api-manager_1   | 	at sun.security.provider.certpath.BasicChecker.check(BasicChecker.java:147)
api-manager_1   | 	at sun.security.provider.certpath.PKIXMasterCertPathValidator.validate(PKIXMasterCertPathValidator.java:125)
api-manager_1   | 	... 32 more
api-manager_1   | [2018-03-28 21:00:54,907]  WARN - DataEndpointGroup No receiver is reachable at reconnection, will try to reconnect every 30 sec
api-manager_1   | [2018-03-28 21:00:54,912] ERROR - DataEndpointConnectionWorker Thrift exception
api-manager_1   | org.wso2.carbon.databridge.agent.exception.DataEndpointAuthenticationException: Thrift exception
api-manager_1   | 	at org.wso2.carbon.databridge.agent.endpoint.thrift.ThriftDataEndpoint.login(ThriftDataEndpoint.java:49)
api-manager_1   | 	at org.wso2.carbon.databridge.agent.endpoint.DataEndpointConnectionWorker.connect(DataEndpointConnectionWorker.java:93)
api-manager_1   | 	at org.wso2.carbon.databridge.agent.endpoint.DataEndpointConnectionWorker.run(DataEndpointConnectionWorker.java:42)
api-manager_1   | 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
api-manager_1   | 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
api-manager_1   | 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
api-manager_1   | 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
api-manager_1   | 	at java.lang.Thread.run(Thread.java:748)
api-manager_1   | Caused by: org.apache.thrift.transport.TTransportException: javax.net.ssl.SSLException: Connection has been shutdown: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: signature check failed
api-manager_1   | 	at org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:161)
api-manager_1   | 	at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:65)
api-manager_1   | 	at org.wso2.carbon.databridge.commons.thrift.service.secure.ThriftSecureEventTransmissionService$Client.send_connect(ThriftSecureEventTransmissionService.java:104)
api-manager_1   | 	at org.wso2.carbon.databridge.commons.thrift.service.secure.ThriftSecureEventTransmissionService$Client.connect(ThriftSecureEventTransmissionService.java:95)
api-manager_1   | 	at org.wso2.carbon.databridge.agent.endpoint.thrift.ThriftDataEndpoint.login(ThriftDataEndpoint.java:45)
api-manager_1   | 	... 7 more
api-manager_1   | Caused by: javax.net.ssl.SSLException: Connection has been shutdown: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: signature check failed
api-manager_1   | 	at sun.security.ssl.SSLSocketImpl.checkEOF(SSLSocketImpl.java:1551)
api-manager_1   | 	at sun.security.ssl.SSLSocketImpl.checkWrite(SSLSocketImpl.java:1563)
api-manager_1   | 	at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:71)
api-manager_1   | 	at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
api-manager_1   | 	at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
api-manager_1   | 	at org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:159)
api-manager_1   | 	... 11 more
api-manager_1   | Caused by: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: signature check failed
api-manager_1   | 	at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
api-manager_1   | 	at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1959)
api-manager_1   | 	at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:328)
api-manager_1   | 	at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:322)
api-manager_1   | 	at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1614)
api-manager_1   | 	at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216)
api-manager_1   | 	at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052)
api-manager_1   | 	at sun.security.ssl.Handshaker.process_record(Handshaker.java:987)
api-manager_1   | 	at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1072)
api-manager_1   | 	at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1385)
api-manager_1   | 	at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:757)
api-manager_1   | 	at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:123)
api-manager_1   | 	... 14 more
api-manager_1   | Caused by: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: signature check failed
api-manager_1   | 	at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:362)
api-manager_1   | 	at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:270)
api-manager_1   | 	at sun.security.validator.Validator.validate(Validator.java:260)
api-manager_1   | 	at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324)
api-manager_1   | 	at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:229)
api-manager_1   | 	at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:124)
api-manager_1   | 	at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1596)
api-manager_1   | 	... 21 more
api-manager_1   | Caused by: java.security.cert.CertPathValidatorException: signature check failed
api-manager_1   | 	at sun.security.provider.certpath.PKIXMasterCertPathValidator.validate(PKIXMasterCertPathValidator.java:135)
api-manager_1   | 	at sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:223)
api-manager_1   | 	at sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:140)
api-manager_1   | 	at sun.security.provider.certpath.PKIXCertPathValidator.engineValidate(PKIXCertPathValidator.java:79)
api-manager_1   | 	at java.security.cert.CertPathValidator.validate(CertPathValidator.java:292)
api-manager_1   | 	at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:357)
api-manager_1   | 	... 27 more
api-manager_1   | Caused by: java.security.SignatureException: Signature length not correct: got 256 but was expecting 128
api-manager_1   | 	at sun.security.rsa.RSASignature.engineVerify(RSASignature.java:189)
api-manager_1   | 	at java.security.Signature$Delegate.engineVerify(Signature.java:1223)
api-manager_1   | 	at java.security.Signature.verify(Signature.java:656)
api-manager_1   | 	at sun.security.x509.X509CertImpl.verify(X509CertImpl.java:444)
api-manager_1   | 	at sun.security.provider.certpath.BasicChecker.verifySignature(BasicChecker.java:166)
api-manager_1   | 	at sun.security.provider.certpath.BasicChecker.check(BasicChecker.java:147)
api-manager_1   | 	at sun.security.provider.certpath.PKIXMasterCertPathValidator.validate(PKIXMasterCertPathValidator.java:125)
api-manager_1   | 	... 32 more

jeremy303 avatar Mar 28 '18 22:03 jeremy303

@HolySamosa Did you do any changes to the original deployment or is this an Out-of-the-box run ? Please do let us know steps to reproduce since we are not experiencing this issue at our side.

DilanUA avatar Mar 30 '18 01:03 DilanUA

Thanks, @DilanUA.

This is APIM-ISasKM-with-Analytics out-of-the-box-- almost. I did build the docker images locally using jdk1.8.0_161 and mysql-connector-java-5.1.46-bin.jar and modified the docker-compose.yml to pull the local images. Otherwise, no changes.

jeremy303 avatar Mar 30 '18 21:03 jeremy303

@HolySamosa Could you try with a older JDK version, older than _151?

chamilad avatar Apr 02 '18 06:04 chamilad

@HolySamosa and @chamilad Is there any solution for this issue?

ichwill100 avatar May 08 '18 13:05 ichwill100

Is this issue still Open ? I am also getting same error in my WSO2 APIM server for analtyics.

SureshG02 avatar Dec 19 '18 11:12 SureshG02

am getting the same error... my setup apim-m 2.5.0 dockerized api-m admin + gateway docker on one machine and apim-analytics server on another.

** using OpenJDK 8 with AllowAll for hostname verification

enabled ssl debug and seeing the following in the api-m (client) logs...

trigger seeding of SecureRandom done seeding SecureRandom [2019-01-04 23:30:08,634] ERROR - DataEndpointConnectionWorker Error while trying to connect to the endpoint. Cannot borrow client for ssl://10.204.131.28:7714 org.wso2.carbon.databridge.agent.exception.DataEndpointAuthenticationException: Cannot borrow client for ssl://10.204.131.28:7714 at org.wso2.carbon.databridge.agent.endpoint.DataEndpointConnectionWorker.connect(DataEndpointConnectionWorker.java:136) at org.wso2.carbon.databridge.agent.endpoint.DataEndpointConnectionWorker.run(DataEndpointConnectionWorker.java:59) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.wso2.carbon.databridge.agent.exception.DataEndpointSecurityException: Error while trying to connect to ssl://10.204.131.28:7714 at org.wso2.carbon.databridge.agent.endpoint.thrift.ThriftSecureClientPoolFactory.createClient(ThriftSecureClientPoolFactory.java:81) at org.wso2.carbon.databridge.agent.client.AbstractClientPoolFactory.makeObject(AbstractClientPoolFactory.java:39) at org.apache.commons.pool.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:1212) at org.wso2.carbon.databridge.agent.endpoint.DataEndpointConnectionWorker.connect(DataEndpointConnectionWorker.java:126) ... 6 more Caused by: org.apache.thrift.transport.TTransportException: Could not connect to 10.204.131.28 on port 7714 at org.apache.thrift.transport.TSSLTransportFactory.createClient(TSSLTransportFactory.java:237) at org.apache.thrift.transport.TSSLTransportFactory.getClientSocket(TSSLTransportFactory.java:169) at org.wso2.carbon.databridge.agent.endpoint.thrift.ThriftSecureClientPoolFactory.createClient(ThriftSecureClientPoolFactory.java:64) ... 9 more Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:589) at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:673) at sun.security.ssl.SSLSocketImpl.(SSLSocketImpl.java:432) at sun.security.ssl.SSLSocketFactoryImpl.createSocket(SSLSocketFactoryImpl.java:88) at org.apache.thrift.transport.TSSLTransportFactory.createClient(TSSLTransportFactory.java:233) ... 11 more

  • apim-analytics server logs that it's listening on Thrift receiver started on 0.0.0.0:7714 Thrift receiver started on 0.0.0.0:7614

I can ping to the ip 10.204.131.28 from the client api-m.

Any prompt help is really appreciated...

Thanks

tekatool avatar Jan 04 '19 23:01 tekatool

the same issue for me, any solution.

vmonsanto avatar May 29 '19 14:05 vmonsanto

@HolySamosa Did you import the certificate of Api Manager? I think is necessary to do that for the communication between this solutions. So remember that the user to authenticate api manager with analytics is admin and the password too.

ximeraz avatar Jul 14 '20 23:07 ximeraz