Exposing invalid certificates error metrics

[ppatierno]

While using the strimzi oauth on a specific listener of a Kafka broker but due to an invalid certificate, I am getting exceptions like this one:

java.lang.RuntimeException: Failed to fetch public keys needed to validate JWT signatures: https://<ADDRESS-HERE>/certs
	at io.strimzi.kafka.oauth.validator.JWTSignatureValidator.fetchKeys(JWTSignatureValidator.java:266)
	at io.strimzi.kafka.oauth.validator.JWTSignatureValidator.lambda$new$0(JWTSignatureValidator.java:174)
	at io.strimzi.kafka.oauth.validator.BackOffTaskScheduler$RunnableTask.run(BackOffTaskScheduler.java:142)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)
	at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:349)
	at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:292)
	at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:287)
	at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1356)
	at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1231)
	at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1174)
	at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392)
	at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:443)
	at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:421)
	at java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:182)
	at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:171)
	at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1418)
	at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1324)
	at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:440)
	at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:411)
	at java.base/sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:567)
	at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:197)
	at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:168)
	at io.strimzi.kafka.oauth.common.HttpUtil.request(HttpUtil.java:132)
	at io.strimzi.kafka.oauth.common.HttpUtil.request(HttpUtil.java:86)
	at io.strimzi.kafka.oauth.common.HttpUtil.get(HttpUtil.java:50)
	at io.strimzi.kafka.oauth.validator.JWTSignatureValidator.fetchKeys(JWTSignatureValidator.java:256)
	... 8 more

The only way to catch the error is when a client tries to connect and authentication fails on its side but if using a listener with no oauth it's able to connect, so brokers work ok. It would be useful to have the strimzi oauth exposing some useful metrics and specifically in this case error metrics related to this kind of problems so that it's possible to monitor the Kafka cluster and catch the problem in this way firing alerts (thinking about Prometheus + Grafana solution).

Because on the Kafka broker, Strimzi already have JMX exporter to export JMX Kafka metrics to Prometheus, instead of adding another Prometheus endpoint (by the strimzi oauth) it could be better having the plugin to expose metrics in JMX format and then leaving the JMX exporter making them available in Prometheus on the same HTTP endpoint it's used today for scraping.