Incorrect keystore password

[ebaron]

I encountered this bug, where the keystore is created by cert-manager, but the password is incorrect:

INFO: Local save path for flight recordings set as /opt/cryostat.d/recordings.d
Exception in thread "main" io.vertx.core.VertxException: java.io.IOException: keystore password was incorrect
	at io.vertx.core.net.impl.SSLHelper.createContext(SSLHelper.java:336)
	at io.vertx.core.net.impl.SSLHelper.getContext(SSLHelper.java:511)
	at io.vertx.core.net.impl.SSLHelper.validate(SSLHelper.java:536)
	at io.vertx.core.http.impl.HttpServerImpl.listen(HttpServerImpl.java:284)
	at io.vertx.core.http.impl.HttpServerImpl.listen(HttpServerImpl.java:205)
	at io.vertx.core.http.impl.HttpServerImpl.listen(HttpServerImpl.java:186)
	at io.cryostat.net.HttpServer.start(HttpServer.java:108)
	at io.cryostat.Cryostat.main(Cryostat.java:77)
Caused by: java.io.IOException: keystore password was incorrect
	at java.base/sun.security.pkcs12.PKCS12KeyStore.engineLoad(PKCS12KeyStore.java:2117)
	at java.base/sun.security.util.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:222)
	at java.base/java.security.KeyStore.load(KeyStore.java:1479)
	at io.vertx.core.net.impl.KeyStoreHelper.loadKeyStoreOptions(KeyStoreHelper.java:222)
	at io.vertx.core.net.KeyStoreOptionsBase.getHelper(KeyStoreOptionsBase.java:148)
	at io.vertx.core.net.KeyStoreOptionsBase.getKeyManagerFactory(KeyStoreOptionsBase.java:166)
	at io.vertx.core.net.impl.SSLHelper.getKeyMgrFactory(SSLHelper.java:341)
	at io.vertx.core.net.impl.SSLHelper.createContext(SSLHelper.java:294)
	... 7 more
Caused by: java.security.UnrecoverableKeyException: failed to decrypt safe contents entry: javax.crypto.BadPaddingException: Given final block not properly padded. Such issues can arise if a bad key is used during decryption.
	... 15 more

Looking at the secrets in my namespace, it appears that the TLS secrets created by cert-manager were left behind by an old deployment:

$ oc get secret
NAME                                                TYPE                                  DATA   AGE
cryostat-operator-service-account-dockercfg-v5zb9   kubernetes.io/dockercfg               1      11m
cryostat-operator-service-account-token-6vfsv       kubernetes.io/service-account-token   4      11m
cryostat-operator-service-account-token-9b2pn       kubernetes.io/service-account-token   4      11m
cryostat-sample-ca                                  kubernetes.io/tls                     3      21h
cryostat-sample-dockercfg-vbvnn                     kubernetes.io/dockercfg               1      10m
cryostat-sample-grafana-basic                       Opaque                                2      10m
cryostat-sample-grafana-tls                         kubernetes.io/tls                     3      21h
cryostat-sample-jmx-auth                            Opaque                                2      10m
cryostat-sample-keystore                            Opaque                                1      10m
cryostat-sample-tls                                 kubernetes.io/tls                     4      21h
cryostat-sample-token-kgv4x                         kubernetes.io/service-account-token   4      10m
cryostat-sample-token-mxdlh                         kubernetes.io/service-account-token   4      10m

I suspect this was caused by setting spec.enableCertManager to false before the certificates became ready. This prevented the operator from setting the Cryostat CR as owner of the secrets.

[ebaron]

It might be a good idea to check for the certificate/secret objects and delete them if cert-manager is disabled.