Cluster Status Report unable to render after restarting the GoCD server, cluster profile params returned in plain text

[brandonvin]

Summary of issue

After restarting the GoCD server, the cluster status report is unable to render. security_token and kubernetes_cluster_ca_cert attributes on the cluster profile seem to be returned from the plugin as plain text (instead of resolved via secret parameters).

Not sure if this issue is caused by the kubernetes-elastic-agent plugin or the GoCD server itself, but using kubernetes-elastic-agent plugin was the first time I saw this.

Environment

• GoCD 21.1.0
• kubernetes-elastic-agent 3.8.2-338

Steps to reproduce

1. Set up a Kubernetes cluster profile and elastic profile using the plugin. In particular, this includes values for security_token and kubernetes_cluster_ca_cert in the cluster profile
2. After saving the cluster profile, can navigate to Admin > Elastic Agent Configuraiton page, and expand the details of the cluster profile. The values of security_token and kubernetes_cluster_ca_cert are redacted (replaced with ****** etc.)
3. Everything works at this point. Can run pipelines using the plugin and view the cluster status report and agent status report.
4. Restart the GoCD server.
5. Navigate to Admin > Elastic Agents Configuration and look at the cluster profile created earlier.

Expected:

• Opening the Status Report for the Kubernetes cluster profile works the same as it did before restarting the GoCD server
• Opening the Cluster Profile details, the values of security_token and kubernetes_cluster_ca_cert look the same as before restarting the GoCD server (redacted)

Actual:

• Opening the Status Report for the cluster profile shows an error message:

  • I looked at logs and have copied relevant logs below

• Opening the Cluster Profile details, the values of security_token and kubernetes_cluster_ca_cert are shown in plain text

Workaround:

• I was able to get the cluster profile back to the “expected” state by just clicking through the cluster profile Edit dialog once, making no edits, and saving it again. That's an easy workaround, but it would be nice not to have this extra step whenever the server is restarted (to update plugins, update server configs, etc.).

Relevant logs

In the GoCD server logs:

2023-01-18 13:49:22,332 ERROR [192@MessageListener for ServerPingListener] JMSMessageListenerAdapter:87 - Exception thrown in message handling by listener com.thoughtworks.go.server.messaging.elasticagents.ServerPingListener@7770a52
java.lang.RuntimeException: The plugin sent a response that could not be understood by Go. Plugin returned with code '500' and the following response: 'Failed to handle request cd.go.elastic-agent.server-ping'
        at com.thoughtworks.go.plugin.access.PluginRequestHelper.submitRequest(PluginRequestHelper.java:58)
        at com.thoughtworks.go.plugin.access.elastic.v5.ElasticAgentExtensionV5.serverPing(ElasticAgentExtensionV5.java:151)
        at com.thoughtworks.go.plugin.access.elastic.ElasticAgentExtension.serverPing(ElasticAgentExtension.java:63)
        at com.thoughtworks.go.plugin.access.elastic.ElasticAgentPluginRegistry.serverPing(ElasticAgentPluginRegistry.java:51)
        at com.thoughtworks.go.server.messaging.elasticagents.ServerPingListener.onMessage(ServerPingListener.java:31)
        at com.thoughtworks.go.server.messaging.elasticagents.ServerPingListener.onMessage(ServerPingListener.java:21)
        at com.thoughtworks.go.server.messaging.activemq.JMSMessageListenerAdapter.runImpl(JMSMessageListenerAdapter.java:83)
        at com.thoughtworks.go.server.messaging.activemq.JMSMessageListenerAdapter.run(JMSMessageListenerAdapter.java:63)
        at java.base/java.lang.Thread.run(Unknown Source)

In the kubernetes-elastic-agents plugin logs:

2023-01-18 13:49:22,332 ERROR [192@MessageListener for ServerPingListener] KubernetesPlugin:127 - Failed to handle request cd.go.elastic-agent.server-ping
io.fabric8.kubernetes.client.KubernetesClientException: An error has occurred.
        at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:103)
        at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:97)
        at io.fabric8.kubernetes.client.utils.HttpClientUtils.applyCommonConfiguration(HttpClientUtils.java:214)
        at io.fabric8.kubernetes.client.okhttp.OkHttpClientFactory.createHttpClient(OkHttpClientFactory.java:89)
        at io.fabric8.kubernetes.client.utils.HttpClientUtils.createHttpClient(HttpClientUtils.java:164)
        at io.fabric8.kubernetes.client.BaseClient.<init>(BaseClient.java:48)
        at io.fabric8.kubernetes.client.BaseKubernetesClient.<init>(BaseKubernetesClient.java:159)
        at io.fabric8.kubernetes.client.DefaultKubernetesClient.<init>(DefaultKubernetesClient.java:42)
        at cd.go.contrib.elasticagent.KubernetesClientFactory.createClientFor(KubernetesClientFactory.java:83)
        at cd.go.contrib.elasticagent.KubernetesClientFactory.client(KubernetesClientFactory.java:60)
        at cd.go.contrib.elasticagent.KubernetesAgentInstances.unregisteredAfterTimeout(KubernetesAgentInstances.java:201)
        at cd.go.contrib.elasticagent.KubernetesAgentInstances.terminateUnregisteredInstances(KubernetesAgentInstances.java:126)
        at cd.go.contrib.elasticagent.executors.ServerPingRequestExecutor.performCleanupForACluster(ServerPingRequestExecutor.java:63)
        at cd.go.contrib.elasticagent.executors.ServerPingRequestExecutor.execute(ServerPingRequestExecutor.java:48)
        at cd.go.contrib.elasticagent.KubernetesPlugin.handle(KubernetesPlugin.java:91)
        at com.thoughtworks.go.plugin.infra.DefaultPluginManager.lambda$submitTo$0(DefaultPluginManager.java:134)
        at com.thoughtworks.go.plugin.infra.FelixGoPluginOSGiFramework.executeActionOnTheService(FelixGoPluginOSGiFramework.java:208)
        at com.thoughtworks.go.plugin.infra.FelixGoPluginOSGiFramework.doOn(FelixGoPluginOSGiFramework.java:164)
        at com.thoughtworks.go.plugin.infra.DefaultPluginManager.submitTo(DefaultPluginManager.java:131)
        at com.thoughtworks.go.plugin.access.PluginRequestHelper.submitRequest(PluginRequestHelper.java:49)
        at com.thoughtworks.go.plugin.access.elastic.v5.ElasticAgentExtensionV5.serverPing(ElasticAgentExtensionV5.java:151)
        at com.thoughtworks.go.plugin.access.elastic.ElasticAgentExtension.serverPing(ElasticAgentExtension.java:63)
        at com.thoughtworks.go.plugin.access.elastic.ElasticAgentPluginRegistry.serverPing(ElasticAgentPluginRegistry.java:51)
        at com.thoughtworks.go.server.messaging.elasticagents.ServerPingListener.onMessage(ServerPingListener.java:31)
        at com.thoughtworks.go.server.messaging.elasticagents.ServerPingListener.onMessage(ServerPingListener.java:21)
        at com.thoughtworks.go.server.messaging.activemq.JMSMessageListenerAdapter.runImpl(JMSMessageListenerAdapter.java:83)
        at com.thoughtworks.go.server.messaging.activemq.JMSMessageListenerAdapter.run(JMSMessageListenerAdapter.java:63)
        at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.security.cert.CertificateException: Could not parse certificate: java.io.IOException: Empty input
        at java.base/sun.security.provider.X509Factory.engineGenerateCertificate(Unknown Source)
        at java.base/java.security.cert.CertificateFactory.generateCertificate(Unknown Source)
        at io.fabric8.kubernetes.client.internal.CertUtils.createTrustStore(CertUtils.java:102)
        at io.fabric8.kubernetes.client.internal.CertUtils.createTrustStore(CertUtils.java:75)
        at io.fabric8.kubernetes.client.internal.SSLUtils.trustManagers(SSLUtils.java:140)
        at io.fabric8.kubernetes.client.internal.SSLUtils.trustManagers(SSLUtils.java:90)
        at io.fabric8.kubernetes.client.utils.HttpClientUtils.applyCommonConfiguration(HttpClientUtils.java:203)
        ... 25 common frames omitted
Caused by: java.io.IOException: Empty input
        ... 32 common frames omitted