parse CA file with --tls_certificate in bazel does not work

[jiaquan1]

Following the instruction https://github.com/bazelbuild/bazel-buildfarm/blob/main/_site/docs/configuration/authentication.md to set up the tls certification, configured the server with

sslCertificatePath: /vault/secrets/server.crt
sslPrivateKeyPath: /vault/secrets/server.pem

On the client repo, --tls_certificate=config/security/test_certs/bf_ca.crt added in bazel cmd, the path is relative path to bazel workspace. Keep getting the following error

ERROR: javax.net.ssl.SSLHandshakeException: General OpenSslEngine problem
        at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.handshakeException(ReferenceCountedOpenSslEngine.java:1898)
        at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.wrap(ReferenceCountedOpenSslEngine.java:822)
        at java.base/javax.net.ssl.SSLEngine.wrap(Unknown Source)
        at io.netty.handler.ssl.SslHandler.wrap(SslHandler.java:1040)
        at io.netty.handler.ssl.SslHandler.wrapNonAppData(SslHandler.java:926)
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1408)
        at io.netty.handler.ssl.SslHandler.unwrapNonAppData(SslHandler.java:1326)
        at io.netty.handler.ssl.SslHandler.access$1800(SslHandler.java:169)
        at io.netty.handler.ssl.SslHandler$SslTasksRunner.resumeOnEventExecutor(SslHandler.java:1717)
        at io.netty.handler.ssl.SslHandler$SslTasksRunner.access$2000(SslHandler.java:1608)
        at io.netty.handler.ssl.SslHandler$SslTasksRunner$2.run(SslHandler.java:1766)
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469)
        at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base/java.lang.Thread.run(Unknown Source)
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at java.base/sun.security.validator.PKIXValidator.doBuild(Unknown Source)
        at java.base/sun.security.validator.PKIXValidator.engineValidate(Unknown Source)
        at java.base/sun.security.validator.Validator.validate(Unknown Source)
        at java.base/sun.security.ssl.X509TrustManagerImpl.validate(Unknown Source)
        at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(Unknown Source)
        at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(Unknown Source)
        at io.netty.handler.ssl.ReferenceCountedOpenSslClientContext$ExtendedTrustManagerVerifyCallback.verify(ReferenceCountedOpenSslClientContext.java:234)
        at io.netty.handler.ssl.ReferenceCountedOpenSslContext$AbstractCertificateVerifier.verify(ReferenceCountedOpenSslContext.java:748)
        at io.netty.internal.tcnative.CertificateVerifierTask.runTask(CertificateVerifierTask.java:36)
        at io.netty.internal.tcnative.SSLTask.run(SSLTask.java:48)
        at io.netty.internal.tcnative.SSLTask.run(SSLTask.java:42)
        at io.netty.handler.ssl.ReferenceCountedOpenSslEngine$TaskDecorator.run(ReferenceCountedOpenSslEngine.java:1465)
        at io.netty.handler.ssl.SslHandler$SslTasksRunner.run(SslHandler.java:1784)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        ... 1 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at java.base/sun.security.provider.certpath.SunCertPathBuilder.build(Unknown Source)
        at java.base/sun.security.provider.certpath.SunCertPathBuilder.engineBuild(Unknown Source)
        at java.base/java.security.cert.CertPathBuilder.build(Unknown Source)
        ... 16 more

ERROR: Failed to query remote execution capabilities: General OpenSslEngine problem

Confirmed the CA file is -rw-rw-r--, and the server.crt and server.pem on the server pod is -rw-r--r-- , Copied the server.crt from the server pod to local machine, confirmed the openssl verify with CA file oK. openssl s_client TLS handshake with server pod 443 port was successful. On server log, just outbound and inbound after buildfarm successfully started

javax.net.ssl|ALL|07|grpc-nio-worker-ELG-3-26|2024-04-22 22:11:55.632 UTC|SSLEngineImpl.java:825|Closing outbound of SSLEngine
javax.net.ssl|ALL|07|grpc-nio-worker-ELG-3-26|2024-04-22 22:11:55.633 UTC|SSLEngineImpl.java:786|Closing inbound of SSLEngine

[jiaquan1]

@luxe need more help to make https://github.com/bazelbuild/bazel-buildfarm/pull/1251 work

[werkt]

Reading this, it looks like a bazel client-sourced exception not on connection to buildfarm, but on cert load, which this project would have no ability to affect. Recommend you post over at github.com/bazelbuild/bazel if it is indeed local to bazel.