helpdesk icon indicating copy to clipboard operation
helpdesk copied to clipboard

Published 20 hours ago • verified publisher icon jenkins-infra

Downloads /latest directory out of date

Open simonh-mmw opened this issue 3 years ago • 27 comments

Service(s)

get.jenkins.io

Summary

Is there a reason why the /latest download does not match the currently latest LTS of 2.346.1?

https://get.jenkins.io/war-stable/latest/

Fixed URLs are really useful for a number of reasons, and if /latest is no longer to be used, should it not to removed?

Reproduction steps

  1. Latest published version https://get.jenkins.io/war-stable/2.346.1/jenkins.war
  2. Compare to https://get.jenkins.io/war-stable/latest/jenkins.war

simonh-mmw avatar Jul 05 '22 13:07 simonh-mmw

For the latest links to work you need to use the update center which has logic for that.

Weekly: https://updates.jenkins.io/latest/jenkins.war

Not sure exactly what the LTS url is, the script is here though: https://github.com/jenkins-infra/update-center2/blob/master/site/generate.sh

timja avatar Jul 05 '22 14:07 timja

Thanks for the pointer to https://updates.jenkins.io @timja . The link to latest LTS is https://updates.jenkins.io/stable/latest/ with the jenkins.war at https://updates.jenkins.io/stable/latest/jenkins.war

MarkEWaite avatar Jul 05 '22 18:07 MarkEWaite

This does raise though that ATH is pointing to get.Jenkins.io in run.sh and if that that’s not working we should swap to the update center, let’s keep it open to check that

timja avatar Jul 05 '22 19:07 timja

Quick check before nighty night:

  • On archives.jenkins.io, the "latest" link seems ok: https://archives.jenkins.io/war-stable/latest/ (datetime of the war looks good).
  • Same on the OSUSL mirrors: https://ftp-nyc.osuosl.org/pub/jenkins/war-stable/latest/ and https://ftp-chi.osuosl.org/pub/jenkins/war-stable/latest/

The only "culprit" is get.jenkins.io itself which has its own "fallback" file server.

dduportal avatar Jul 05 '22 19:07 dduportal

On get.jenkins.io's file system (took one of the 3 "fileserver" pod randomly:

root@htdocs/war-stable# ls -l latest/
total 64896
-rwxrwxrwx 1 1000 1000 66452488 Aug 17  2020 jenkins.war
-rwxrwxrwx 1 1000 1000       77 Aug 17  2020 jenkins.war.sha256

while it is asymlink on archives.jenkins.io, or on updates.jenkins.io

$ ls -l latest
lrwxrwxrwx 1 www-data www-data 7 Jul  5 18:00 latest -> 2.346.1

=> either we should "delete" the directory latest/ and see if the "sync script" takes care of copying the correct symlink, or we might need to create this symlink and make sure that whatever script used for sync updates it

dduportal avatar Jul 05 '22 19:07 dduportal

As a general fact, there is none of the latest links on get.jenkins.io which are valid.

dduportal avatar Jul 06 '22 09:07 dduportal

Temporarly removed the war-stable/latest directory to check wether or not the update_center script synchronizes the symlink.

=> https://get.jenkins.io/war-stable/latest/ is HTTP/404 for now!

dduportal avatar Jul 06 '22 09:07 dduportal

I raised https://github.com/jenkinsci/acceptance-test-harness/pull/812 on ATH

timja avatar Jul 06 '22 09:07 timja

I raised jenkinsci/acceptance-test-harness#812 on ATH

Thinking aloud there but this ATh change should only be temporarly until we fix the issue. get.jenkins.io should still be considered as the default download source.

dduportal avatar Jul 06 '22 09:07 dduportal

Update: https://get.jenkins.io/war-stable/latest/ is now showing the same content as the reference servers and mirrors 🥳

  • Step 1: deletion of the latest directory in the mirrorbits azure file bucket (mounted in R/W in the mirrorbits pods)
  • Step 2: wait for the update_center job (on trusted.ci) which runs freqeuntly (Every 5 min roughly) to finish with success and synchronize files.

Next step: I'll perform the same one-time operation to one of the plugins to see if it also fixes the issue

dduportal avatar Jul 06 '22 10:07 dduportal

Will it work next time though 👅

timja avatar Jul 06 '22 10:07 timja

Will it work next time though 👅

Good question: the filesystem does NOT have a symlink now, but a directory. I assume that whatever rsync / blobxfer command which does the synchronization is dereferencing the symlinks.

Let's check on plugins: easier to trigger a new release and see the result.

dduportal avatar Jul 06 '22 10:07 dduportal

Plugin test:

  • https://get.jenkins.io/plugins/jenkins-infra-test/latest/ => 2020-12-09 date
  • Filesystem on get.jenkins.io (mirrorbits):
repo/plugins/jenkins-infra-test$ ls -l
total 0
drwxrwxrwx 2 mirrorbits mirrorbits 0 Nov 16  2020 1.0
drwxrwxrwx 2 mirrorbits mirrorbits 0 Dec  1  2020 1.1
drwxrwxrwx 2 mirrorbits mirrorbits 0 Mar 25  2021 12.vb659278afc6b
drwxrwxrwx 2 mirrorbits mirrorbits 0 Mar 25  2021 13.v50f44d6f3875
drwxrwxrwx 2 mirrorbits mirrorbits 0 Mar 25  2021 25.vc6471c550e70
drwxrwxrwx 2 mirrorbits mirrorbits 0 Mar 25  2021 37.v560be292b3e4
drwxrwxrwx 2 mirrorbits mirrorbits 0 Mar 25  2021 42.v19b576f53ae3
drwxrwxrwx 2 mirrorbits mirrorbits 0 Mar 25  2021 44.vb770b1e25577
drwxrwxrwx 2 mirrorbits mirrorbits 0 Aug 17  2021 48.v44dd7301178e
drwxrwxrwx 2 mirrorbits mirrorbits 0 Aug 27  2021 50.v756d280f8072
drwxrwxrwx 2 mirrorbits mirrorbits 0 Sep 10  2021 54.vd2265a5fa9a2
drwxrwxrwx 2 mirrorbits mirrorbits 0 Sep 10  2021 56.v40a08369fa8c
drwxrwxrwx 2 mirrorbits mirrorbits 0 Mar 27 15:56 58.vdf0feab90495
drwxrwxrwx 2 mirrorbits mirrorbits 0 Jun  2 21:38 62.v0d4f02c3969f
drwxrwxrwx 2 mirrorbits mirrorbits 0 Jun 11 04:34 64.v21d9d793c858
drwxrwxrwx 2 mirrorbits mirrorbits 0 Jun 13 20:45 69.v49b_5815e8158
drwxrwxrwx 2 mirrorbits mirrorbits 0 Jun 13 22:32 71.vdb_a_65780b_f0a_
drwxrwxrwx 2 mirrorbits mirrorbits 0 Jun 13 23:42 73.v10b_5ff00a_a_d6
drwxrwxrwx 2 mirrorbits mirrorbits 0 Jun 14 00:32 75.v3d8c8a_c0177e
drwxrwxrwx 2 mirrorbits mirrorbits 0 Jun 14 18:32 78.veb_b_67886f8b_9
drwxrwxrwx 2 mirrorbits mirrorbits 0 Dec  9  2020 latest
repo/plugins/jenkins-infra-test$ ls -l latest/
total 6
-rwxrwxrwx 1 mirrorbits mirrorbits 5127 Dec  9  2020 jenkins-infra-test.hpi
  • https://archives.jenkins.io/plugins/jenkins-infra-test/latest/ and https://ftp-nyc.osuosl.org/pub/jenkins/plugins/jenkins-infra-test/latest/ show a date 2022-06-14.

dduportal avatar Jul 06 '22 10:07 dduportal

  • Just deleted the latest directory for the plugin jenkins-infra-test plugin.
  • https://get.jenkins.io/plugins/jenkins-infra-test/latest/ is now HTTP/404.
  • Let's wait for update center to fix it (at least one time).

=> then: we'll trigger a new release of the plugin and see what happens.

dduportal avatar Jul 06 '22 10:07 dduportal

FTR https://updates.jenkins.io/latest/jenkins.war is currently returning 404 as it's pointing to https://get.jenkins.io/war/2.358/jenkins.war which doesn't exist yet

timja avatar Jul 06 '22 10:07 timja

  • Just deleted the latest directory for the plugin jenkins-infra-test plugin.

    • https://get.jenkins.io/plugins/jenkins-infra-test/latest/ is now HTTP/404.

    • Let's wait for update center to fix it (at least one time).

=> then: we'll trigger a new release of the plugin and see what happens.

  • https://get.jenkins.io/plugins/jenkins-infra-test/latest/ is now OK with a datetime at 2022-07-06 10:16
  • Next step: let's trigger a new plugin release (@timja what is process: should I open a new dummy PR on it and once merged it will release? Or is there another way?) for https://github.com/jenkinsci/jenkins-infra-test-plugin

dduportal avatar Jul 06 '22 10:07 dduportal

FTR https://updates.jenkins.io/latest/jenkins.war is currently returning 404 as it's pointing to https://get.jenkins.io/war/2.358/jenkins.war which doesn't exist yet

That is ... unexpected 🤔

dduportal avatar Jul 06 '22 10:07 dduportal

Next step: let's trigger a new plugin release (@timja what is process: should I open a new dummy PR on it and once merged it will release? Or is there another way?) for jenkinsci/jenkins-infra-test-plugin

Yes do that, make sure you label it with enhancement

timja avatar Jul 06 '22 10:07 timja

FTR https://updates.jenkins.io/latest/jenkins.war is currently returning 404 as it's pointing to https://get.jenkins.io/war/2.358/jenkins.war which doesn't exist yet

It's another issue: sounds like the weekl packaging process is stuck. Just received pagerdutty alerts about this.

dduportal avatar Jul 06 '22 10:07 dduportal

FTR https://updates.jenkins.io/latest/jenkins.war is currently returning 404 as it's pointing to https://get.jenkins.io/war/2.358/jenkins.war which doesn't exist yet

https://github.com/jenkins-infra/helpdesk/issues/3037

dduportal avatar Jul 06 '22 10:07 dduportal

  • https://github.com/jenkinsci/jenkins-infra-test-plugin/pull/31 for the "plugin" latest check
  • https://updates.jenkins.io/latest/jenkins.war is ok again (#3037 finished)

dduportal avatar Jul 06 '22 11:07 dduportal

OK, so confirmation that, once the latest/ directory exists, then the "sync script" does not try to update it on the azurefile bucket:

[latest/](https://get.jenkins.io/plugins/jenkins-infra-test/latest/)                 2022-07-06 10:16    -   
[80.v441ea_08a_25b_e/](https://get.jenkins.io/plugins/jenkins-infra-test/80.v441ea_08a_25b_e/)    2022-07-06 13:05    -   
[78.veb_b_67886f8b_9/](https://get.jenkins.io/plugins/jenkins-infra-test/78.veb_b_67886f8b_9/)    2022-06-14 18:32    -

=> It means that we should not repeat the manual operation that I did earlier on the production data volume, but instead update the script of the update_center repository so that it overrides the latest directories when uploading to get.jenkins.io

dduportal avatar Jul 06 '22 15:07 dduportal

Had to deep on the machine pkg.origin.jenkins.io history since it was manually managed for months before we put it back under puppet management. The goal is to understand "what process" is in charge of syncing the recent changes to get.jenkins.io's files.

  • There is a crontab defined for the (to be removed soon) user mirrorbrain that runs once per hour
    • (legacy) definition can be found here: https://github.com/jenkins-infra/jenkins-infra/blob/f42f1c06ea1ebd96256d0b417f7278405f4e59b7/dist/profile/manifests/mirrorbrain.pp#L301-L308
    • On real life it is defined as the following:
# Puppet Name: mirrorbrain-sync-releases
0 * * * * cd /srv/releases && ./sync.sh --full-sync > last_sync.log

  • The sync.sh script that is in production can be found here: https://github.com/jenkins-infra/mirror-scripts/blob/master/sync.sh.

    • In particular, we have a "full" sync once per hour (due to the crontab): https://github.com/jenkins-infra/mirror-scripts/blob/master/sync.sh#L71-L76 which uses the blobxfer command line to upload data to the remote Azurefile bucket served by get.jenkins.io
    • This script is also called during the Jenkins Core process, but without the --full-sync flag: https://github.com/jenkins-infra/release/blob/master/utils/release.bash#L539
    • The symlink latest for the Jenkins Core Version seems to also be managed by this script: https://github.com/jenkins-infra/release/blob/master/utils/release.bash#L539 by calling the script https://github.com/jenkins-infra/mirror-scripts/blob/master/update-latest-symlink.sh with eventually a flag per Core baseline (weekly, rc, etc.).

  • The last_sync.log analysis shows that the blobxfer transfer is explictly NOT overwriting the latest directories and files (among other files). This beahvior is because of the --no-overwrite flag defined in the call to blobxfer in the script: https://github.com/jenkins-infra/mirror-scripts/blob/master/sync.sh#L75

$ grep 'latest' last_sync.log | head -n20
>> Updating the latest symlink for weekly
>> Updating the latest symlink for weekly RC
>> Updating the latest symlink for LTS
>> Updating the latest symlink for LTS RC
2022-07-07 16:02:17,154 INFO - not overwriting remote file: mirrorbits/war-stable/latest/jenkins.war (local: /srv/releases/jenkins/war-stable/latest/jenkins.war)
2022-07-07 16:02:17,173 INFO - not overwriting remote file: mirrorbits/war-stable/latest/jenkins.war.sha256 (local: /srv/releases/jenkins/war-stable/latest/jenkins.war.sha256)
2022-07-07 16:02:57,462 INFO - not overwriting remote file: mirrorbits/war/latest/jenkins.war (local: /srv/releases/jenkins/war/latest/jenkins.war)
2022-07-07 16:02:57,480 INFO - not overwriting remote file: mirrorbits/war/latest/jenkins.war.sha256 (local: /srv/releases/jenkins/war/latest/jenkins.war.sha256)
2022-07-07 16:03:26,270 INFO - not overwriting remote file: mirrorbits/war-stable-rc/latest/jenkins.war (local: /srv/releases/jenkins/war-stable-rc/latest/jenkins.war)
2022-07-07 16:03:26,288 INFO - not overwriting remote file: mirrorbits/war-stable-rc/latest/jenkins.war.sha256 (local: /srv/releases/jenkins/war-stable-rc/latest/jenkins.war.sha256)
2022-07-07 16:03:28,979 INFO - not overwriting remote file: mirrorbits/war-rc/latest/jenkins.war (local: /srv/releases/jenkins/war-rc/latest/jenkins.war)
2022-07-07 16:03:36,088 INFO - not overwriting remote file: mirrorbits/plugins/gitlab-branch-source/latest/gitlab-branch-source.hpi (local: /srv/releases/jenkins/plugins/gitlab-branch-source/latest/gitlab-branch-source.hpi)
2022-07-07 16:03:36,593 INFO - not overwriting remote file: mirrorbits/plugins/appray/latest/appray.hpi (local: /srv/releases/jenkins/plugins/appray/latest/appray.hpi)
2022-07-07 16:03:36,651 INFO - not overwriting remote file: mirrorbits/plugins/disable-github-multibranch-status/latest/disable-github-multibranch-status.hpi (local: /srv/releases/jenkins/plugins/disable-github-multibranch-status/latest/disable-github-multibranch-status.hpi)
2022-07-07 16:03:36,738 INFO - not overwriting remote file: mirrorbits/plugins/r/latest/r.hpi (local: /srv/releases/jenkins/plugins/r/latest/r.hpi)
2022-07-07 16:03:36,876 INFO - not overwriting remote file: mirrorbits/plugins/nant/latest/nant.hpi (local: /srv/releases/jenkins/plugins/nant/latest/nant.hpi)
2022-07-07 16:03:37,031 INFO - not overwriting remote file: mirrorbits/plugins/policycenter-gate-validator/latest/policycenter-gate-validator.hpi (local: /srv/releases/jenkins/plugins/policycenter-gate-validator/latest/policycenter-gate-validator.hpi)
2022-07-07 16:03:37,150 INFO - not overwriting remote file: mirrorbits/plugins/deployed-on-column/latest/deployed-on-column.hpi (local: /srv/releases/jenkins/plugins/deployed-on-column/latest/deployed-on-column.hpi)
2022-07-07 16:03:37,674 INFO - not overwriting remote file: mirrorbits/plugins/tasks/latest/tasks.hpi (local: /srv/releases/jenkins/plugins/tasks/latest/tasks.hpi)
2022-07-07 16:03:39,599 INFO - not overwriting remote file: mirrorbits/plugins/github/latest/github.hpi (local: /srv/releases/jenkins/plugins/github/latest/github.hpi)


$ grep 'latest' last_sync.log | wc -l
2198

$ grep 'latest' last_sync.log | grep -v '/plugins/'
>> Updating the latest symlink for weekly
>> Updating the latest symlink for weekly RC
>> Updating the latest symlink for LTS
>> Updating the latest symlink for LTS RC
2022-07-07 16:02:17,154 INFO - not overwriting remote file: mirrorbits/war-stable/latest/jenkins.war (local: /srv/releases/jenkins/war-stable/latest/jenkins.war)
2022-07-07 16:02:17,173 INFO - not overwriting remote file: mirrorbits/war-stable/latest/jenkins.war.sha256 (local: /srv/releases/jenkins/war-stable/latest/jenkins.war.sha256)
2022-07-07 16:02:57,462 INFO - not overwriting remote file: mirrorbits/war/latest/jenkins.war (local: /srv/releases/jenkins/war/latest/jenkins.war)
2022-07-07 16:02:57,480 INFO - not overwriting remote file: mirrorbits/war/latest/jenkins.war.sha256 (local: /srv/releases/jenkins/war/latest/jenkins.war.sha256)
2022-07-07 16:03:26,270 INFO - not overwriting remote file: mirrorbits/war-stable-rc/latest/jenkins.war (local: /srv/releases/jenkins/war-stable-rc/latest/jenkins.war)
2022-07-07 16:03:26,288 INFO - not overwriting remote file: mirrorbits/war-stable-rc/latest/jenkins.war.sha256 (local: /srv/releases/jenkins/war-stable-rc/latest/jenkins.war.sha256)
2022-07-07 16:03:28,979 INFO - not overwriting remote file: mirrorbits/war-rc/latest/jenkins.war (local: /srv/releases/jenkins/war-rc/latest/jenkins.war)
2022-07-07 16:17:05,317 INFO - not overwriting remote file: mirrorbits/windows/latest (local: /srv/releases/jenkins/windows/latest)
2022-07-07 16:17:20,527 INFO - not overwriting remote file: mirrorbits/windows-stable/latest (local: /srv/releases/jenkins/windows-stable/latest)

dduportal avatar Jul 07 '22 16:07 dduportal

  • The blobxfer command line is the version 1.6.0 of 2019 and is installed (manually) with PyPi on the machine (to be tracked in https://github.com/jenkins-infra/helpdesk/issues/2970):
$ pip list | grep blobxfer 
blobxfer (1.6.0)
$ blobxfer  --version
blobxfer, version 1.6.0
$ dpkg -l | grep -c blobxfer # No package
0

dduportal avatar Jul 07 '22 16:07 dduportal

  • The documentation for the 1.6.0 blobxfer version does not mention anything about symlink, neither their issue tracker (ref. https://blobxfer.readthedocs.io/en/1.6.0/10-cli-usage/#upload and https://github.com/Azure/blobxfer).
  • Neither the latest 1.11.0 version: https://blobxfer.readthedocs.io/en/1.11.0/10-cli-usage/.

Ping @olblak @daniel-beck @timja do you have any memory if blobxfer supports symlinks? Wondering because if it does not, then will have to add a specific blobxfer command to check the latest md5 with a force-override

dduportal avatar Jul 07 '22 16:07 dduportal

Also, as per #3100 (closed as duplicate), @jnord raised the following issue:

Summary

According to the timestamps https://get.jenkins.io/war/latest/jenkins.war was last updated in 2020 This is not the case and if you download the file you can see it is 2.364 (which is the latest at the time of writing). However this causes confusion as it appears it has not been updated.

Reproduction steps:

go to https://get.jenkins.io/war/ Notice the modification data on latest (possibly ok if this is a directory and the children are symlink)s. go to https://get.jenkins.io/war/latest/ Notice the modification date on the files is still in 2020 Expected result, the timestamps shown in the web interface correspond to (approximatly) the time of publishing of the war.

dduportal avatar Aug 19 '22 11:08 dduportal

if blobxfer supports symlinks?

azcopy supports them: https://docs.microsoft.com/en-us/azure/storage/common/storage-ref-azcopy-copy

--follow-symlinks Follow symbolic links when uploading from local file system.

lemeurherve avatar Sep 06 '22 13:09 lemeurherve