Race condition causing jobs not to be loaded on Jenkins startup

[bjaglin]

Running latest master (and latest stable release), jobs with a gitlab merge request trigger on a https server sometimes disappear after a Jenkins restart, because of this exception (and despite the fact that the ignoreCertificateErrors flag is enabled).

SEVERE: Failed Loading job service-rtb-mr
java.lang.Error: javax.net.ssl.SSLHandshakeException: You can disable certificate checking by setting ignoreCertificateErrors on GitlabHTTPRequestor
        at org.gitlab.api.http.GitlabHTTPRequestor$1.fetch(GitlabHTTPRequestor.java:199)
        at org.gitlab.api.http.GitlabHTTPRequestor$1.hasNext(GitlabHTTPRequestor.java:154)
        at org.gitlab.api.GitlabAPI.getAllProjects(GitlabAPI.java:93)
        at org.jenkinsci.plugins.gitlab.GitlabRepository.getProjectForPath(GitlabRepository.java:98)
        at org.jenkinsci.plugins.gitlab.GitlabRepository.checkState(GitlabRepository.java:39)
        at org.jenkinsci.plugins.gitlab.GitlabRepository.init(GitlabRepository.java:30)
        at org.jenkinsci.plugins.gitlab.GitlabMergeRequestBuilder.build(GitlabMergeRequestBuilder.java:52)
        at org.jenkinsci.plugins.gitlab.GitlabBuildTrigger.start(GitlabBuildTrigger.java:49)
        at org.jenkinsci.plugins.gitlab.GitlabBuildTrigger.start(GitlabBuildTrigger.java:28)
        at hudson.model.AbstractProject.onLoad(AbstractProject.java:331)
        at hudson.model.Project.onLoad(Project.java:90)
        at hudson.model.Items.load(Items.java:276)
        at jenkins.model.Jenkins$18.run(Jenkins.java:2589)
        at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169)
        at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:282)
        at jenkins.model.Jenkins$7.runTask(Jenkins.java:899)
        at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:210)
        at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
Caused by: javax.net.ssl.SSLHandshakeException: You can disable certificate checking by setting ignoreCertificateErrors on GitlabHTTPRequestor
        at org.gitlab.api.http.GitlabHTTPRequestor.parse(GitlabHTTPRequestor.java:275)
        at org.gitlab.api.http.GitlabHTTPRequestor.access$200(GitlabHTTPRequestor.java:31)
        at org.gitlab.api.http.GitlabHTTPRequestor$1.fetch(GitlabHTTPRequestor.java:192)
        ... 20 more

Probably a race condition between GitlabBuildTrigger#GitlabBuildTriggerDescriptor#configure() and GitlabBuildTrigger#GitlabBuildTriggerDescriptor#getGitlab() ? As a workaround, setting the initial value of _ignoreCertificateErrors to true fixes the issue...

[bjaglin]

Actually, my workaround doesn't seem to work every time, so there is definitely an annoying race condition. Maybe earlier, during config loading?

[kamazee]

I'm also affected by this issue, but don't have enough experience to debug it. I'd be grateful if anyone can advice anything. Logging (even with messages added here and there around the issue) doesn't help much -- Java Gitlab API seems to get true for ignoring certificate errors, but raises an exception anyway.

However, there's another way out which is probably more secure than ignoring certificate errors: adding a certificate to a storage. This is a fairly simple operation:

1. Fetch a certificate (Here is a guide)
2. Find out where your Java installation keeps certificate information (/etc/ssl/certs/java/cacerts for Debian)
3. # keytool -import -file RETRIEVED_CERT -alias SENSIBLE_ALIAS -keystore CERT_STORAGE

[djdefi]

I attempted to import the cert via keytool, however still recieve the error. Maybe I did something wrong there, will try again when I get a moment.

[kamazee]

@rtrauntvein did it work? It might require restarting Jenkins.

[chrode]

You should check https://github.com/timols/jenkins-gitlab-merge-request-builder-plugin/issues/87#issuecomment-59209892 That solution might be valid too.

[dmaslakov]

Issue still exists even with option 'Ignore SSL Certificate Errors' on. Are there plans to get it fixed?

SEVERE: Failed Loading job merge-build-job
java.lang.Error: javax.net.ssl.SSLHandshakeException: You can disable certificate checking by setting ignoreCertificateErrors on GitlabHTTPRequestor. SSL Error: sun.security.validator.ValidatorException: PKIX path building failed: sun.se
curity.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at org.gitlab.api.http.GitlabHTTPRequestor$1.fetch(GitlabHTTPRequestor.java:218)
        at org.gitlab.api.http.GitlabHTTPRequestor$1.hasNext(GitlabHTTPRequestor.java:174)
        at org.gitlab.api.http.GitlabHTTPRequestor.getAll(GitlabHTTPRequestor.java:143)
        at org.gitlab.api.GitlabAPI.getProjects(GitlabAPI.java:168)
        at org.jenkinsci.plugins.gitlab.GitlabRepository.getProjectForPath(GitlabRepository.java:98)
        at org.jenkinsci.plugins.gitlab.GitlabRepository.checkState(GitlabRepository.java:39)
        at org.jenkinsci.plugins.gitlab.GitlabRepository.init(GitlabRepository.java:30)
        at org.jenkinsci.plugins.gitlab.GitlabMergeRequestBuilder.build(GitlabMergeRequestBuilder.java:52)
        at org.jenkinsci.plugins.gitlab.GitlabBuildTrigger.start(GitlabBuildTrigger.java:44)
        at org.jenkinsci.plugins.gitlab.GitlabBuildTrigger.start(GitlabBuildTrigger.java:27)
        at hudson.model.AbstractProject.onLoad(AbstractProject.java:326)
        at hudson.model.Project.onLoad(Project.java:95)
        at hudson.model.Items.load(Items.java:279)
        at jenkins.model.Jenkins$17.run(Jenkins.java:2673)
        at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169)
        at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:282)
        at jenkins.model.Jenkins$7.runTask(Jenkins.java:903)
        at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:210)
        at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
Caused by: javax.net.ssl.SSLHandshakeException: You can disable certificate checking by setting ignoreCertificateErrors on GitlabHTTPRequestor. SSL Error: sun.security.validator.ValidatorException: PKIX path building failed: sun.security
.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at org.gitlab.api.http.GitlabHTTPRequestor.parse(GitlabHTTPRequestor.java:294)
        at org.gitlab.api.http.GitlabHTTPRequestor.access$200(GitlabHTTPRequestor.java:39)
        at org.gitlab.api.http.GitlabHTTPRequestor$1.fetch(GitlabHTTPRequestor.java:211)
        ... 21 more

[chrode]

Did you try switching to openjdk?

[dmaslakov]

No I did not. If I get it correctly, openjdk will solve the issue with certificate validation due to different cipher length. But it's likely not mine case because after Jenkins was restarted, all tasks were successfully loaded. Looks more like race condition when loading system configuration.

[jpappe]

For whatever it's worth, this hit me after updating my Jenkins server and all plugins to their latest releases:

• Jenkins: 1.598
• GitLab Merge Request Builder: 1.2.2
• OpenJDK

Several of my jobs failed to load. I restarted Jenkins and a different (smaller) set of jobs failed to load. I then reloaded config from disk and now all my jobs are back.