Update Jira LTS from 9.12.x to 10.3.x

[NotMyFault]

Service(s)

Jira

Summary

issues.jenkins.io runs on 9.12.x being EOL by November this year.

I propose to update to 10.3.x, that's the next supported LTS. @MarkEWaite would you be so kind and submit a ticket to the LFX infra, as we depend on their availability?

Similar to https://github.com/jenkins-infra/helpdesk/issues/3939 like last year.

Reproduction steps

No response

[MarkEWaite]

I have submitted https://jira.linuxfoundation.org/plugins/servlet/desk/portal/2/IT-28124 to the Linux Foundation team. My phrasing in that request was:

As noted in https://github.com/jenkins-infra/helpdesk/issues/4644 , the Jenkins Jira installation at https://issues.jenkins.io/ is running a Jira version that will reach end of life before the end of calendar 2025.

We need the Linux Foundation team to upgrade Jenkins Jira to the most recent Jira Data Center version. We believe that is currently a 10.3.x version.

[MarkEWaite]

The Linux Foundation ticketing system shows that the ticket has been assigned to their team member that has performed our previous updates.

[MarkEWaite]

Latest status from Linux Foundation:

The upgrade to Jira 10.3.5 LTS involves moving to version 17 of the Java JDK. That version of Java is not available on the existing VM instance. So this upgrade will require a new VM.

Much of the work and testing can be done without any impact on the existing Jira service, so this may not be any more disruptive than a regular upgrade. However, I will need time to create the new environment. After that is done, I can work with you to schedule the maintenance window that will put the 10.3.5 LTS Jira into service.

I'll work with you to schedule a maintenance window when I have the new system ready.

[MarkEWaite]

Latest status shared by our contact at Linux Foundation is:

Yes, this is taking a long time as this system is getting rebuilt during this upgrade.

A new instance has been built with RHEL9 and the required Java17. An ansible role has been identified to manage this installation. The development of the ansible configuration management needs to be completed. After that, a migration plan for the database, license, plugins, and attachments needs to be tested. When that is done, there will be a test url that I'll ask you to look through for validation that it performs as expected. Once the new instance passes validation, we can schedule the cutover which will involve enough downtime to sync data and switch the test and production url. I'll then decommission the old system.

It may be a few more weeks before I'm ready to share the test url.

[dduportal]

As per a discussion with @MarkEWaite:

• LF support team is asking if we create a new DNS record which would point to their new VM for testing purpose. The propose record is testissues.jenkins.io. They gave us the target to put on the CNAME record
• We are going to ask them for IP restriction only on our VPN to reach this instance as a matter a safety. We have to add the VPN route (client and server side) for this to happen
• Mark asked LF to add @dduportal in the LF issue tracker (As he already has an account)

[dduportal]

Update:

• DNS created for the LF:

$ dig +short testissues.jenkins.io               
jira-rhel9-alb-1841417744.us-west-2.elb.amazonaws.com.
52.88.217.28 

• VPN is now routing requests to this IP:

$ netstat -rn | grep '52.88.217.28'                                                                                                    
52.88.217.28/32    10.9.0.1           UGSc                utun6   

• I have been granted access to the LF issue and communicated with them
  • The DNS
  • The need for them to share their outbound IPs to allow in our LDAP
  • The request to restrict access to the test instance through the VPN for safety

[dduportal]

Update: discussion in progress with Ryan at the LF (in the JIRA ticket) to exchange securely the LDAP password (I requested his GPG key so we can transmit it securely without requiring a 3rd party service) .

[dduportal]

Resuming work as publick8s is now migrated to a new location.

First step: I now have access to the LDAP as admin using Apache Directory Studio. Runbook update: https://github.com/jenkins-infra/runbooks/pull/108

Next step is to create a new binding user for Ryan at LF and share the user/password with him

[dduportal]

Update: commented in the LF issue (https://jira.linuxfoundation.org/plugins/servlet/desk/portal/2/IT-28124) with the new user's DN and password (GPG encrypted to Ryan):

Hello '@'Ryan, back at this again now we have migrated our LDAP.

Since I failed to find the current "redacted" password (at least it is secured) unencrypted, I've created a new bind user "redacted" to unblock you on the tests.

The full DN ans the password are specified in the attached text file (encrypted with GPG using your public key with ID "redacted" (expiration in redacted).

Let me know if you can:

1. Decrypt the attached file to access the user DN and password
2. If yes, then is it working to bind the test instance to LDAP? (I've tested the authentication with this new account using Apache Directory Studio as a preliminary so if it fails for you then we'll check what are the differences).

Note: We'll change the current's "redacted" password to the one of "redacted" in the LDAP when you'll perform the final migration of JIRA and we'll get rid of the test user at that moment.

[timja]

You need to put backticks around @ mentions when copying text into GitHub otherwise you'll notify actual users

[dduportal]

Update: Ryan (from LF) has performed with success a complete migration on the test instance. He'll come back to us later this week to plan the final migration.

I'm taking over from Mark as he is in vacations for 2 weeks.

[dduportal]

Proposed maintenance window for the JIRA upgrade: Thursday 13 November starting at 04:00pm UTC (08:00am PST, 09:00 Central time and 05:00pm Paris time) until 10:00pm UTC.

• [x] Agreed with Ryan at LF and already announced at https://status.linuxfoundation.org/incidents/3tc7nb5f8ds2
• Jenkins side announcement:
  • [x] On status.jenkins.io: https://github.com/jenkins-infra/status/commit/439c1f869d2d8d98b923fcb57e9155c797a80f7c
  • [x] On the developer mailing list: https://groups.google.com/g/jenkinsci-dev/c/lLWn1CLqzRM/m/RRS3__vBAAAJ
  • [x] On the jenkins-infra mailing list: https://groups.google.com/g/jenkins-infra/c/J-7sQOI6heY/m/FuMCt763BQAJ
  • [x] On the Matrix jenkinsci/jenkins channel: https://matrix.to/#/!ouJVNKRtaWHFflDvBW:gitter.im/$C0clbMyiup8DEhq-4G1UlD8IG4YMm-gn2BEemzZeEvk?via=gitter.im&via=matrix.org&via=minds.com
  • [x] On the Community Forums: https://community.jenkins.io/t/issues-jenkins-io-jira-lts-migration-on-13-november-2025-at-16h00-utc/35789

[MarkEWaite]

Currently scheduled to upgrade this on Nov 13, 2025. See the status page.

[dduportal]

Temporarily disabling the synthetics Datadog monitors during the migration with https://github.com/jenkins-infra/datadog/pull/324

[rynofinn]

The re-index on the new Jira is complete. I haven't seen any traffic from non-lfadmin accounts. When I hear that tests are passing, I'll move on to migrating the domains in the final cutover.

[dduportal]

The re-index on the new Jira is complete. I haven't seen any traffic from non-lfadmin accounts. When I hear that tests are passing, I'll move on to migrating the domains in the final cutover.

@rynofinn when trying to log-in as dduportal, I get a com.atlassian.crowd.exception.runtime.OperationFailedException error message:

Can you check the logs to see what the problem could be?

[rynofinn]

I'll need to restart jira to troubleshoot the ldap directory server. So expect the service to bounce once or twice.

[rynofinn]

I had the wrong ldap password in Jira. Can you log in now?

[MarkEWaite]

I had the wrong ldap password in Jira. Can you log in now?

I tried to login and it reported that Jira is in recovery mode. My regular username and password were rejected.

[rynofinn]

it should have worked in recovery mode. I'll restart to get back to normal.

[rynofinn]

I have restarted jira to get out of recovery mode. Does your login work now?

[MarkEWaite]

Login works with my username and password. I've verified several different actions in the UI including updating state and adding comments. All looks good to me

[rynofinn]

OK, great! Thanks for covering those tests that are hard for me to do without an LDAP account. I'll move ahead with the cutover from testissues.jenkins.io to issues.jenkins.io

[rynofinn]

I'm also going to extend the maintenance by another hour to make sure I have enough time to complete the transition.

[rynofinn]

The cutover to https://issues.jenkins.io worked. I have to restart the service one more time to switch the name from testissues to issues.

[MarkEWaite]

Thanks. I've updated the maintenance window on status.jenkins.io with this pull request:

• https://github.com/jenkins-infra/status/pull/667

[rynofinn]

https://issues.jenkins.io is up and ready for service. Let me know if you see any problems. I'm going to do some clean up tasks but they will not cause any service disruption. I'll end the service window in 30 minutes and announce that the service is up on statuspage.io.

[rynofinn]

The IP address allow list for issues.jenkins.io has been opened up to the public. There is no longer a rule that only allows access from my address and 52.232.183.117/32

[MarkEWaite]

I've confirmed that I can access https://issues.jenkins.io . I'll update status.jenkins.io with the end of the maintenance window when you let me know that you're done.

[rynofinn]

I'm all done. Please close the maintenance window. Thanks for all your help!