Creates scoring to evaluate plugins requiring old version of Jenkins core

[alecharp]

trafficstars

Description of the bug

Currently, the probe is failing only is the version couldn't be find on the update-center data.

We should evaluate the version we get from the update-center and see when was the version core was released. If that date is older than a threshold (1 year sounds enough), we should mark the probe as failing.

EDIT: have to credit @basil for the idea

[daniel-beck]

I recommend this check be more elaborate to better mirror common motivations for recent core dependencies and our existing documentation, e.g. https://www.jenkins.io/doc/developer/plugin-development/choosing-jenkins-baseline/

I propose something like the following: (for the purposes of these examples we're at 2.387.2 and 2.399 at the moment, with 2.319.3 being the oldest LTS release supported by UC infra)

• It's good (1.0) to have a weekly dependency if that is newer than the latest released LTS line (2.388+).
• It's acceptable (~0.7) to have a weekly dependency if that is newer than the latest historic LTS line (2.376+).
• It's tolerable (~0.5) to have a weekly dependency if that version corresponds to an LTS baseline (e.g., 2.332).
• It's bad to have any other weekly dependency (0.0)
• It's good (1.0) to have an LTS dependency on the current LTS line's .1 or newer (2.387.1, 2.387.2)
• It's good (1.0) to have an LTS dependency on the latest release of the latest historic LTS line (2.375.4)
• It's acceptable (~0.8) to have an LTS dependency on a different release of the latest historic LTS line (e.g., 2.375.2)
• For older LTS lines, consider all of the following:
  • How many implied dependencies are you picking up? They're bad (~10% deduction per dependency or so).
  • Are you older than is supported by the update sites (>13 months old dependency)? That's bad (-30% deduction).
  • Are you depending on a release other than the latest in its line? That's bad (-20% deduction).

So, let's see how this would work with some examples:

• 2.390 - 100%
• 2.387.2 - 100%
• 2.387.1 - 100%
• 2.387 - 70% (corresponds to baseline)
• 2.385 - 50% (between current and historic LTS lines)
• 2.375.4 - 100%
• 2.375.3 - 80%
• 2.375.2 - 80%
• 2.375.1 - 80%
• 2.375 - 70% (baseline)
• 2.370 - 0% (old weekly dependency)
• 2.361.4 - 100%
• 2.361.3 - 80%
• 2.361.2 - 80%
• 2.361.1 - 80%
• 2.346.3 - 90% (2.356 split instance-identity)
• 2.346.2 - 70%
• 2.346.1 - 70%
• 2.332.4 - 90%
• 2.332.3 - 70%
• 2.332.2 - 70%
• 2.332.1 - 70%
• 2.319.3 - 70% (2.330 split two plugins)
• 2.319.2 - 50%
• 2.319.1 - 20% (no longer supported by infra)
• 2.303.3 - 50%
• 2.303.2 - 30%
• 2.303.1 - 30%
• 2.289.3 - 50%
• 2.289.2 - 30%
• 2.289.1 - 30%
• 2.277.4 - 10% (2.281 split sshd)
• 2.277.3 - 0%
• 2.277.2 - 0%
• 2.277.1 - 0%
• 2.263.4 - 10%
• 2.263.x - 0%
• 2.249.3 - 10%
• …
• 2.190.3 - 10%
• 2.190.x - 0%
• 2.176.4 - 0% (latest in its line, but has now picked up another implied dependency – from now on all are 0%)

While there's a huge difference between 2.277.4 and 2.190.3 (~2 years), as of today it doesn't make any practical difference between them unless we start considering special one-off circumstances like the tables-to-divs redesign as well. Even then, no difference between 2.263.4 and 2.190.3, so they should score alike.

Obviously this can be done more or less aggressive for each of the factors influencing dependencies on older historic LTS lines.

Another way to look at this might be to take usage stats into account, e.g., deduct from the score if your version is older than what X% of Jenkins users are using. That might be more difficult to justify though?

[alecharp]

In order to implement that, we should keep the current behavior of the probe (fetch and persisting the jenkins.version value) but the scoring (to be implementing, see #243) should have a fine grade implementation to use that degrading scoring as you described.

This is totally possible.