Issues in concurrent accessing Trust-Store-temp.jks in the GW startup

Open NishanthiWi opened this issue 1 year ago • 0 comments

Description

While migrating the APIM-2.6.0 to APIM-4.0.0, the Trust-Store-temp.jks is getting corrupted with the following error after updating the trust store with a few set of certificates.

TID: [-1234] [] [2024-03-06 16:51:59,580] INFO {org.wso2.carbon.apimgt.impl.certificatemgt.CertificateManagerImpl} - The Transport Sender will be re-initialized in few minutes. TID: [-1234] [] [2024-03-06 16:51:59,580] INFO {org.wso2.carbon.apimgt.impl.certificatemgt.CertificateManagerImpl} - The certificate with Alias 'china-dev-ingestion_8' is successfully added to the Gateway Trust Store. TID: [-1234] [] [2024-03-06 16:51:59,708] INFO {org.wso2.carbon.apimgt.impl.certificatemgt.CertificateManagerImpl} - The Transport Sender will be re-initialized in few minutes. TID: [-1234] [] [2024-03-06 16:51:59,708] INFO {org.wso2.carbon.apimgt.impl.certificatemgt.CertificateManagerImpl} - The certificate with Alias 'china-qa-consumption_8' is successfully added to the Gateway Trust Store. TID: [-1234] [] [2024-03-06 16:51:59,765] INFO {org.wso2.carbon.apimgt.impl.certificatemgt.CertificateManagerImpl} - The Transport Sender will be re-initialized in few minutes. TID: [-1234] [] [2024-03-06 16:51:59,765] INFO {org.wso2.carbon.apimgt.impl.certificatemgt.CertificateManagerImpl} - The certificate with Alias 'china-qa-ingestion_8' is successfully added to the Gateway Trust Store. TID: [-1234] [] [2024-03-06 16:51:59,790] ERROR {org.wso2.carbon.apimgt.impl.utils.CertificateMgtUtils} - Error in loading the certificate. java.io.EOFException at java.base/java.io.DataInputStream.readInt(DataInputStream.java:397) at java.base/sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:665) at java.base/sun.security.util.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:222) at java.base/java.security.KeyStore.load(KeyStore.java:1479) at org.wso2.carbon.apimgt.impl.utils.CertificateMgtUtils.addCertificateToTrustStore_aroundBody8(CertificateMgtUtils.java:185) at org.wso2.carbon.apimgt.impl.utils.CertificateMgtUtils.addCertificateToTrustStore(CertificateMgtUtils.java:168) at org.wso2.carbon.apimgt.impl.utils.CertificateMgtUtils.addCertificateToSenderTrustStore_aroundBody2(CertificateMgtUtils.java:115) at org.wso2.carbon.apimgt.impl.utils.CertificateMgtUtils.addCertificateToSenderTrustStore(CertificateMgtUtils.java:111) at org.wso2.carbon.apimgt.impl.certificatemgt.CertificateManagerImpl.addCertificateToListenerOrSenderProfile_aroundBody16(CertificateManagerImpl.java:254) at org.wso2.carbon.apimgt.impl.certificatemgt.CertificateManagerImpl.addCertificateToListenerOrSenderProfile(CertificateManagerImpl.java:239) at org.wso2.carbon.apimgt.impl.certificatemgt.CertificateManagerImpl.addCertificateToGateway_aroundBody10(CertificateManagerImpl.java:202) at org.wso2.carbon.apimgt.impl.certificatemgt.CertificateManagerImpl.addCertificateToGateway(CertificateManagerImpl.java:196) at org.wso2.carbon.apimgt.gateway.EndpointCertificateDeployer.retrieveCertificatesAndDeploy_aroundBody6(EndpointCertificateDeployer.java:114) at org.wso2.carbon.apimgt.gateway.EndpointCertificateDeployer.retrieveCertificatesAndDeploy(EndpointCertificateDeployer.java:98) at org.wso2.carbon.apimgt.gateway.EndpointCertificateDeployer.deployCertificatesAtStartup_aroundBody0(EndpointCertificateDeployer.java:67) at org.wso2.carbon.apimgt.gateway.EndpointCertificateDeployer.deployCertificatesAtStartup(EndpointCertificateDeployer.java:62) at org.wso2.carbon.apimgt.gateway.listeners.GatewayStartupListener.lambda$5_aroundBody44(GatewayStartupListener.java:346) at org.wso2.carbon.apimgt.gateway.listeners.GatewayStartupListener.lambda$5(GatewayStartupListener.java:346) at java.base/java.lang.Thread.run(Thread.java:829)

TID: [-1234] [] [2024-03-06 16:51:59,797] INFO {org.wso2.carbon.apimgt.impl.certificatemgt.CertificateManagerImpl} - The Transport Sender will be re-initialized in few minutes. TID: [-1234] [] [2024-03-06 16:51:59,798] ERROR {org.wso2.carbon.apimgt.impl.certificatemgt.CertificateManagerImpl} - Error adding the certificate with Alias 'estimator-api-dev.cbre.com_9' to the Gateway Trust Store TID: [-1234] [] [2024-03-06 16:51:59,821] INFO {org.wso2.carbon.apimgt.impl.certificatemgt.CertificateManagerImpl} - The Transport Sender will be re-initialized in few minutes. TID: [-1234] [] [2024-03-06 16:51:59,821] INFO {org.wso2.carbon.apimgt.impl.certificatemgt.CertificateManagerImpl} - The certificate with Alias 'china-uat-consumption_8' is successfully added to the Gateway Trust Store.

Please note that the, eager loading is enabled on the APIM servers and, there are more than 20 tenants. As this is a migration, the backend certificates are not added from the publisher. The trust store should be updated with the available certificates in the DB.

Steps to Reproduce

Enable tenant eager loading
Create more than 5 tenants and more than 60 APIs per tenant with backend certificates
Start the GW

Affected Component

Analytics

Version

4.0.0.265

Environment Details (with versions)

No response

Relevant Log Output

No response

Related Issues

(https://github.com/wso2-enterprise/wso2-apim-internal/issues/5719)

Suggested Labels

No response

Apr 02 '24 08:04 NishanthiWi