helm-charts icon indicating copy to clipboard operation
helm-charts copied to clipboard

Published 20 hours ago • verified publisher icon jenkinsci

When httpsKeyStore is enabled, what additional changes are needed?

Open lknite opened this issue 2 years ago • 3 comments

Describe the bug

After configuring httpsKeyStore in the helm file the jenkins installation can no longer be accessed.

Version of Helm and Kubernetes

- Helm:
version.BuildInfo{Version:"v3.9.0", GitCommit:"7ceeda6c585217a19a1131663d8cd1f7d641b2a7", GitTreeState:"clean", GoVersion:"go1.17.5"}

- Kubernetes: 
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.1", GitCommit:"3ddd0f45aa91e2f30c70734b175631bec5b5825a", GitTreeState:"clean", BuildDate:"2022-05-24T12:26:19Z", GoVersion:"go1.18.2", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.4
Server Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.0", GitCommit:"4ce5a8954017644c5420bae81d72b09b735c21f0", GitTreeState:"clean", BuildDate:"2022-05-03T13:38:19Z", GoVersion:"go1.18.1", Compiler:"gc", Platform:"linux/amd64"}

Chart version

jenkins-4.1.8

What happened?

After seeing certificate related issues in the log files I attempted to fix this by configuring the local on prem certificate authority as a keystore.  I already have experience doing this with keycloak, which had the same Java-related error and truststore solution.

However, after adding the trustore (which I was able to verify by execing in and listing out the contents of the truststore and providing the correct password), jenkins is not coming back up.  The service is targeting port 8080 but I think it needs to instead target 8081.  I tried setting the controller.targetPort value, but get that the address 0.0.0.0 has already been bound.

Could someone assist me in the follow-up steps needed after enabling a truststore via the helm chart?  I checked for notes around the section in the helm file but didn't see any "additional steps required".  Not sure if this is a bug or if there are just additional steps.

log w/ PKIX error indicating that a truststore is needed:

Retrieving update center information
Update center URL: https://updates.jenkins.io/update-center.json?version=2.346.1
Cache miss for: update-center-2.346.1
Unable to retrieve JSON from https://updates.jenkins.io/update-center.json?version=2.346.1: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
Unable to retrieve JSON from https://updates.jenkins.io/update-center.json?version=2.346.1: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
io.jenkins.tools.pluginmanager.impl.UpdateCenterInfoRetrievalException: Error getting update center json

What you expected to happen?

to attempt to configure oic again and this time it work because the truststore has the right ca-bundle

How to reproduce it

1. use helm to install jenkins (include oic-auth module in values.yaml)
2. pause a moment while cert-manager installs certificates using onprem certificate authority and registers ip via external-dns
3. login and verify everything works
4. configure oic-auto and notice PKIX related error messages in the log
5. add httpsKeyStore section:

    httpsKeyStore:
      enable: true
      password: <passwordgoeshere>
      jenkinsKeyStoreBase64Encoded: MIIEswIB...

Anything else we need to know?

Am using a subchart, hence the top level 'jenkins:'.

values.yaml

jenkins:

  persistence:
    enabled: true

  controller:
    installPlugins:
    - kubernetes:3600.v144b_cd192ca_a_
    - workflow-aggregator:581.v0c46fa_697ffd
    - git:4.11.3
    - configuration-as-code:1429.v09b_044a_c93de 
    - oic-auth:1.8

    ingress:
      enabled: true
      ingressClassName: nginx
      hostName: jenkins.prod.k.home.net
      tls:
      - secretName: jenkins.prod.k.home.net-tls
        hosts:
        - jenkins.prod.k.home.net
      annotations:
        cert-manager.io/issuer: "cluster-adcs-issuer"                   #use specific name of issuer
        cert-manager.io/issuer-kind: "ClusterAdcsIssuer"                #or ClusterAdcsIssuer
        cert-manager.io/issuer-group: "adcs.certmanager.csf.nokia.com"
        nginx.ingress.kubernetes.io/rewrite-target: /
        nginx.ingress.kubernetes.io/proxy-body-size: 1000m

#    servicePort: 8080
#    targetPort: 8081
    httpsKeyStore:
      enable: true
      password: <passwordgoeshere>
      jenkinsKeyStoreBase64Encoded: MIIEs...

The jenkins server is targeting port 8080 which, if I log into the pod and check seems to be https and it doesn't like an HTTP request being sent to it.

lknite avatar Jun 26 '22 19:06 lknite

If I set 'targetPort: 8081' the container fails with this in the log:

Defaulted container "jenkins" out of: jenkins, config-reload, init (init)
Running from: /usr/share/jenkins/jenkins.war
2022-06-26 20:40:49.872+0000 [id=1]     INFO    org.eclipse.jetty.util.log.Log#initialized: Logging initialized @385ms to org.eclipse.jetty.util.log.JavaUtilLog
2022-06-26 20:40:49.928+0000 [id=1]     INFO    winstone.Logger#logInternal: Beginning extraction from war file
2022-06-26 20:40:49.965+0000 [id=1]     WARNING o.e.j.s.handler.ContextHandler#setContextPath: Empty contextPath
2022-06-26 20:40:50.151+0000 [id=1]     INFO    winstone.Logger#logInternal: Exclude Ciphers [^.*_(MD5|SHA|SHA1)$, ^TLS_RSA_.*$, ^SSL_.*$, ^.*_NULL_.*$, ^.*_anon_.*$]
2022-06-26 20:40:50.155+0000 [id=1]     INFO    org.eclipse.jetty.server.Server#doStart: jetty-9.4.43.v20210629; built: 2021-06-30T11:07:22.254Z; git: 526006ecfa3af7f1a27ef3a288e2bef7ea9dd7e8; jvm 11.0.15+10
2022-06-26 20:40:50.416+0000 [id=1]     INFO    o.e.j.w.StandardDescriptorProcessor#visitServlet: NO JSP Support for /, did not find org.eclipse.jetty.jsp.JettyJspServlet
2022-06-26 20:40:50.448+0000 [id=1]     INFO    o.e.j.s.s.DefaultSessionIdManager#doStart: DefaultSessionIdManager workerName=node0
2022-06-26 20:40:50.448+0000 [id=1]     INFO    o.e.j.s.s.DefaultSessionIdManager#doStart: No SessionScavenger set, using defaults
2022-06-26 20:40:50.448+0000 [id=1]     INFO    o.e.j.server.session.HouseKeeper#startScavenging: node0 Scavenging every 660000ms
2022-06-26 20:40:50.769+0000 [id=1]     INFO    hudson.WebAppMain#contextInitialized: Jenkins home directory: /var/jenkins_home found at: EnvVars.masterEnvVars.get("JENKINS_HOME")
2022-06-26 20:40:50.888+0000 [id=1]     INFO    o.e.j.s.handler.ContextHandler#doStart: Started w.@2fa3be26{Jenkins v2.332.3,/,file:///var/jenkins_cache/war/,AVAILABLE}{/var/jenkins_cache/war}
2022-06-26 20:40:50.947+0000 [id=1]     INFO    o.e.j.server.AbstractConnector#doStart: Started ServerConnector@75d4a5c2{HTTP/1.1, (http/1.1)}{0.0.0.0:8081}
2022-06-26 20:40:50.962+0000 [id=1]     INFO    o.e.j.server.AbstractConnector#doStop: Stopped ServerConnector@75d4a5c2{HTTP/1.1, (http/1.1)}{0.0.0.0:8081}
2022-06-26 20:40:50.963+0000 [id=1]     INFO    o.e.j.server.AbstractConnector#doStop: Stopped ServerConnector@8317c52{SSL, (ssl, http/1.1)}{0.0.0.0:8081}
2022-06-26 20:40:50.963+0000 [id=1]     INFO    o.e.j.server.session.HouseKeeper#stopScavenging: node0 Stopped scavenging
2022-06-26 20:40:50.978+0000 [id=1]     INFO    hudson.WebAppMain#contextDestroyed: Shutting down a Jenkins instance that was still starting up
java.lang.Throwable: reason
        at hudson.WebAppMain.contextDestroyed(WebAppMain.java:386)
        at org.eclipse.jetty.server.handler.ContextHandler.callContextDestroyed(ContextHandler.java:1074)
        at org.eclipse.jetty.servlet.ServletContextHandler.callContextDestroyed(ServletContextHandler.java:584)
        at org.eclipse.jetty.server.handler.ContextHandler.contextDestroyed(ContextHandler.java:1037)
        at org.eclipse.jetty.servlet.ServletHandler.doStop(ServletHandler.java:319)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:94)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.stop(ContainerLifeCycle.java:180)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.doStop(ContainerLifeCycle.java:201)
        at org.eclipse.jetty.server.handler.AbstractHandler.doStop(AbstractHandler.java:108)
        at org.eclipse.jetty.security.SecurityHandler.doStop(SecurityHandler.java:437)
        at org.eclipse.jetty.security.ConstraintSecurityHandler.doStop(ConstraintSecurityHandler.java:423)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:94)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.stop(ContainerLifeCycle.java:180)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.doStop(ContainerLifeCycle.java:201)
        at org.eclipse.jetty.server.handler.AbstractHandler.doStop(AbstractHandler.java:108)
        at org.eclipse.jetty.server.session.SessionHandler.doStop(SessionHandler.java:520)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:94)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.stop(ContainerLifeCycle.java:180)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.doStop(ContainerLifeCycle.java:201)
        at org.eclipse.jetty.server.handler.AbstractHandler.doStop(AbstractHandler.java:108)
        at org.eclipse.jetty.server.handler.ContextHandler.stopContext(ContextHandler.java:1060)
        at org.eclipse.jetty.servlet.ServletContextHandler.stopContext(ServletContextHandler.java:386)
        at org.eclipse.jetty.webapp.WebAppContext.stopWebapp(WebAppContext.java:1454)
        at org.eclipse.jetty.webapp.WebAppContext.stopContext(WebAppContext.java:1420)
        at org.eclipse.jetty.server.handler.ContextHandler.doStop(ContextHandler.java:1114)
        at org.eclipse.jetty.servlet.ServletContextHandler.doStop(ServletContextHandler.java:297)
        at org.eclipse.jetty.webapp.WebAppContext.doStop(WebAppContext.java:547)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:94)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.stop(ContainerLifeCycle.java:180)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.doStop(ContainerLifeCycle.java:201)
        at org.eclipse.jetty.server.handler.AbstractHandler.doStop(AbstractHandler.java:108)
        at org.eclipse.jetty.server.Server.doStop(Server.java:470)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:94)
        at winstone.Launcher.shutdown(Launcher.java:318)
        at winstone.Launcher.<init>(Launcher.java:205)
        at winstone.Launcher.main(Launcher.java:369)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at Main._main(Main.java:304)
        at Main.main(Main.java:108)
2022-06-26 20:40:50.980+0000 [id=1]     INFO    o.e.j.s.handler.ContextHandler#doStop: Stopped w.@2fa3be26{Jenkins v2.332.3,/,null,STOPPED}{/var/jenkins_cache/war}
Exception in thread "Jenkins initialization thread" java.lang.NoClassDefFoundError: hudson/util/HudsonFailedToLoad
        at hudson.WebAppMain$3.run(WebAppMain.java:261)
Caused by: java.lang.ClassNotFoundException: hudson.util.HudsonFailedToLoad
        at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:476)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:589)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
        at org.eclipse.jetty.webapp.WebAppClassLoader.loadClass(WebAppClassLoader.java:538)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
        ... 1 more
2022-06-26 20:40:50.993+0000 [id=1]     INFO    winstone.Logger#logInternal: Jetty shutdown successfully
java.io.IOException: Failed to start Jetty
        at winstone.Launcher.<init>(Launcher.java:194)
        at winstone.Launcher.main(Launcher.java:369)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at Main._main(Main.java:304)
        at Main.main(Main.java:108)
Caused by: java.io.IOException: Failed to bind to 0.0.0.0/0.0.0.0:8081
        at org.eclipse.jetty.server.ServerConnector.openAcceptChannel(ServerConnector.java:349)
        at org.eclipse.jetty.server.ServerConnector.open(ServerConnector.java:310)
        at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:80)
        at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:234)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
        at org.eclipse.jetty.server.Server.doStart(Server.java:401)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
        at winstone.Launcher.<init>(Launcher.java:192)
        ... 7 more
Caused by: java.net.BindException: Address already in use
        at java.base/sun.nio.ch.Net.bind0(Native Method)
        at java.base/sun.nio.ch.Net.bind(Net.java:459)
        at java.base/sun.nio.ch.Net.bind(Net.java:448)
        at java.base/sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:227)
        at java.base/sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:80)
        at org.eclipse.jetty.server.ServerConnector.openAcceptChannel(ServerConnector.java:344)
        ... 14 more
2022-06-26 20:40:50.994+0000 [id=1]     SEVERE  winstone.Logger#logInternal: Container startup failed
java.net.BindException: Address already in use
        at java.base/sun.nio.ch.Net.bind0(Native Method)
        at java.base/sun.nio.ch.Net.bind(Net.java:459)
        at java.base/sun.nio.ch.Net.bind(Net.java:448)
        at java.base/sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:227)
        at java.base/sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:80)
        at org.eclipse.jetty.server.ServerConnector.openAcceptChannel(ServerConnector.java:344)
Caused: java.io.IOException: Failed to bind to 0.0.0.0/0.0.0.0:8081
        at org.eclipse.jetty.server.ServerConnector.openAcceptChannel(ServerConnector.java:349)
        at org.eclipse.jetty.server.ServerConnector.open(ServerConnector.java:310)
        at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:80)
        at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:234)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
        at org.eclipse.jetty.server.Server.doStart(Server.java:401)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
        at winstone.Launcher.<init>(Launcher.java:192)
Caused: java.io.IOException: Failed to start Jetty
        at winstone.Launcher.<init>(Launcher.java:194)
        at winstone.Launcher.main(Launcher.java:369)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at Main._main(Main.java:304)
        at Main.main(Main.java:108)

lknite avatar Jun 26 '22 20:06 lknite

Well, I got this working with the following changes in my values, but it seems a bit all over the place. Is there a more straight-forward way I could be doing this?

    # exposes 8081 which appears with httpsKeyStore
    extraPorts:
    - name: keystorerelated
      port: 8081

    ingress:
      enabled: true
      ingressClassName: nginx
      paths:
      - pathType: ImplementationSpecific
        backend:
          service:
            name: jenkins
            port:
              number: 8081

Also, I noticed when configuring the kubernetes cloud cluster the Jenkins URL must now have the 8081 in it.

lknite avatar Jun 28 '22 02:06 lknite

I'm not sure the best solution here, but seems the above solutions should be enabled automatically if the keystore is configured, or ... if the keystore is configured, instead of everything changing to 8081 without other references to the port changing, just make it the 8080 so everything use it instead, or possibly just add a note to the keystore setting in the values.yaml indicating the follow-up steps necessary to get it to work.

I can't really suggest a pull request on this one because I'm not sure how this was intended to work.

lknite avatar Jun 28 '22 16:06 lknite