Intermittence “PKIX path building failed” issue during Tessera startup

[anthony-yeong-partior]

This issue is similar to the reported under Quorum Ticket 914362.

Tested versions:

1. Quorum: 22.1.1
2. Tessera: 22.1.3

Reproducible steps:

1. Git clone from https://github.com/anthonyyeong18/quorum-examples a. This is a fork from Consensys, with some config changes to enable SSL and updated images on Tessera
2. Run: a. docker-compose up -d
3. Wait for all tessera services to be up and running (using docker ps)
4. Run “docker log ” on all 7 running tessera containers

Expected results:

1. All 7 containers should be showing following logs: 2022-06-27 20:36:49.974 [pool-4-thread-1] INFO c.q.t.p.p.PartyInfoBroadcaster - Started PartyInfo polling round 2022-06-27 20:36:49.975 [pool-4-thread-1] INFO c.q.t.p.p.PartyInfoBroadcaster - Finished PartyInfo polling round 2022-06-27 20:36:54.975 [pool-4-thread-1] INFO c.q.t.p.p.PartyInfoBroadcaster - Started PartyInfo polling round 2022-06-27 20:36:54.976 [pool-4-thread-1] INFO c.q.t.p.p.PartyInfoBroadcaster - Finished PartyInfo polling round 2022-06-27 20:36:59.976 [pool-4-thread-1] INFO c.q.t.p.p.PartyInfoBroadcaster - Started PartyInfo polling round 2022-06-27 20:36:59.977 [pool-4-thread-1] INFO c.q.t.p.p.PartyInfoBroadcaster - Finished PartyInfo polling round 2022-06-27 20:37:04.980 [pool-4-thread-1] INFO c.q.t.p.p.PartyInfoBroadcaster - Started PartyInfo polling round 2022-06-27 20:37:04.981 [pool-4-thread-1] INFO c.q.t.p.p.PartyInfoBroadcaster - Finished PartyInfo polling round 2022-06-27 20:37:09.983 [pool-4-thread-1] INFO c.q.t.p.p.PartyInfoBroadcaster - Started PartyInfo polling round

Results observed:

1. Out of 7 running tessera containers, few of it will have SSL PKIX issue: 2022-06-27 08:22:01.361 [pool-4-thread-1] INFO c.q.t.p.p.PartyInfoBroadcaster - Started PartyInfo polling round 2022-06-27 08:22:01.362 [pool-4-thread-1] INFO c.q.t.p.p.PartyInfoBroadcaster - Finished PartyInfo polling round 2022-06-27 08:22:01.396 [pool-3-thread-7] WARN c.q.t.p.p.PartyInfoBroadcaster - Failed to connect to node https://txmanager7:9000/, due to javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target 2022-06-27 08:22:01.407 [pool-3-thread-2] WARN c.q.t.p.p.PartyInfoBroadcaster - Failed to connect to node https://txmanager2:9000/, due to javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target 2022-06-27 08:22:01.416 [pool-3-thread-10] WARN c.q.t.p.p.PartyInfoBroadcaster - Failed to connect to node https://txmanager6:9000/, due to javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target 2022-06-27 08:22:01.422 [pool-3-thread-6] WARN c.q.t.p.p.PartyInfoBroadcaster - Failed to connect to node https://txmanager3:9000/, due to javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target 2022-06-27 08:22:01.434 [pool-3-thread-8] WARN c.q.t.p.p.PartyInfoBroadcaster - Failed to connect to node https://txmanager4:9000/, due to javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target 2022-06-27 08:22:01.436 [pool-3-thread-4] WARN c.q.t.p.p.PartyInfoBroadcaster - Failed to connect to node https://txmanager5:9000/, due to javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target 2022-06-27 08:22:01.436 [pool-3-thread-1] WARN c.q.t.p.p.PartyInfoBroadcaster - Failed to connect to node https://172.16.239.103:9000/, due to javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target 2022-06-27 08:22:01.440 [pool-3-thread-5] WARN c.q.t.p.p.PartyInfoBroadcaster - Failed to connect to node https://172.16.239.105:9000/, due to javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target 2022-06-27 08:22:01.443 [pool-3-thread-3] WARN c.q.t.p.p.PartyInfoBroadcaster - Failed to connect to node https://txmanager1:9000/, due to javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target 2022-06-27 08:22:01.444 [pool-3-thread-9] WARN c.q.t.p.p.PartyInfoBroadcaster - Failed to connect to node https://172.16.239.101:9000/, due to javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target 2022-06-27 08:22:06.363 [pool-4-thread-1] INFO c.q.t.p.p.PartyInfoBroadcaster - Started PartyInfo polling round 2022-06-27 08:22:06.364 [pool-4-thread-1] INFO c.q.t.p.p.PartyInfoBroadcaster - Finished PartyInfo polling round 2022-06-27 08:22:06.492 [pool-3-thread-8] WARN c.q.t.p.p.PartyInfoBroadcaster - Failed to connect to node https://txmanager4:9000/, due to javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target 2022-06-27 08:22:06.522 [pool-3-thread-7] WARN c.q.t.p.p.PartyInfoBroadcaster - Failed to connect to node https://txmanager7:9000/, due to javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

Notes:

1. Above steps have been tested several times on: a. Local laptop with Windows 11 Pro – WSL2 Ubuntu 20.04 (12 logical cores,32GB RAM) b. GCP VM – Ubuntu 20.04 (4 vCPU,16GB RAM,30GB HD)
2. Issue was “intermittence” as not all containers having PKIX issue. Hence need to docker log all the tessera containers.
3. In the event if no PKIX issue was observed, try the steps again after cleaning up the “dockerData” folder.
4. This is not related to SSL SNI validation, as issue is reproducible even with the validation disabled.

[PPACI]

I'm having exactly the same issue. Tessera 21.10.0 on kubernetes here.

[antonydenyer]

@PPACI @anthonyyeong18 Can you test with tessera:develop to see if the issue has been mitigated?

[frankie-lim-partior]

@antonydenyer tessera:develop is not there. Merged to master?

[antonydenyer]

https://hub.docker.com/layers/quorumengineering/tessera/develop/images/sha256-21f5c9f4670f028e3303796b86eb2474ef3ec9c361db34d03d5a92a9fdf5d3de?context=explore

[anthony-yeong-partior]

@antonydenyer, have tested the latest tessera:develop image on similar docker compose setup.

No longer see the issue after many run. Thanks for the fix.