Restrict latest versions to stable highest releases only for all default repositories

[walterdeboer]

Description

Added a regex to determine unstable releases. Every release in the repository metadata is examined and only stable releases are considered to determine the highest version number, instead of the latest by date.

Before:

After:

Addressed Issue

closes #2500, fixes #513, fixes #1374

Additional Details

Testing is a bit hard since curernt tests need actual HTTP calls. Extracted a few utility methods for verification.

Cargo and Go report highest stable versions. Updated the other repositories to use the utility methods. All repositories now return stable highest versions when found.

Fixed inaccurate published timestamp. It also got updated when a new lower or unstable version was pushed to the repo. The published timestamp is now set only when the latest version in the metadata equals the latest stable version found so we know for sure the published timestamp belongs to that version.

Centralized all version matching in ComponentVersion, added complex version matching to match semver as well as non-semver and different ecosystems, such as debian and ubuntu, that use epoch numbers in versions, or use labels with ''ubuntu" in it. (#1374)

Checklist

• [x] I have read and understand the contributing guidelines
• [ ] This PR fixes a defect, and I have provided tests to verify that the fix is effective
• [x] This PR implements an enhancement, and I have provided tests to verify that it works as intended
• [ ] This PR introduces changes to the database model, and I have added corresponding update logic
• [x] This PR introduces new or alters existing behavior, and I have updated the documentation accordingly

[sonatype-lift[bot]]

🛠 Lift Auto-fix

Some of the Lift findings in this PR can be automatically fixed. You can download and apply these changes in your local project directory of your branch to review the suggestions before committing.[^1]

# Download the patch
curl https://lift.sonatype.com/api/patch/github.com/DependencyTrack/dependency-track/2501.diff -o lift-autofixes.diff

# Apply the patch with git
git apply lift-autofixes.diff

# Review the changes
git diff

Want it all in a single command? Open a terminal in your project's directory and copy and paste the following command:

curl https://lift.sonatype.com/api/patch/github.com/DependencyTrack/dependency-track/2501.diff | git apply

Once you're satisfied, commit and push your changes in your project. [^1]: You can preview the patch by opening the patch URL in the browser.

[walterdeboer]

Thanks @nscuro, no problem! :-)

[walterdeboer]

I'm looking at https://github.com/DependencyTrack/dependency-track/issues/1374 Looks like some more unit testing is needed....

[msymons]

@walterdeboer, following the release of v4.8.0, this PR now has conflicts that need to be resolved before it can be merged. Please can you fix things? Then the PR can be reviewed.

[melba-lopez]

@nscuro there are no merge conflicts and it is updated with the latest changes of base branch.

[melba-lopez]

@walterdeboer can you help resolve the branch conflict?

[nscuro]

@melba-lopez @walterdeboer The conflict just resolved itself after merging #2965 :)

[nscuro]

Thanks again @walterdeboer for the effort here.

Upon further inspection, there are two major things that prevent us from merging this:

• I had multiple orgs reach out indicating that they in fact want to see SNAPSHOT / pre-release versions to be considered as latest. This seems to be the case for primarily internal repositories, e.g. where developers should be notified when a newer SNAPSHOT version of an internal library is available. What this means is that at the very least, the feature implemented in this PR should be optional, potentially even configurable on a per-repository basis.
• This PR touches a very central class, namely ComponentVersion. This class is not only used during repository meta analysis, but also during internal vulnerability analysis. Trying to deal with multiple versioning schemes (Debian, Ubuntu, RPM, etc.) in a single class is highly error-prone and hard to debug. What we're planning to do is to integrate vers (#2826), which includes implementations for ecosystem-specific version matching and comparisons. The plan is to adopt https://github.com/nscuro/versatile for this. Once added, it will be much easier to extend the comparison logic for more versioning schemes.

[walterdeboer]

Thanks for the heads-up @nscuro I'll have a look how to add support for in/excluding pre-releases, i'll get back on that. ComponentVersion definitely deserves a better foundation. Vers(atile) sounds good!

[undeadly]

This PR fixes my major complaint with Dependency Track. Please revive this PR and get it merged.