https://issues.apache.org/jira/browse/SOLR-17406

Description

With Gradle it is possible to manage versions and dependencies in gradle files via version catalogs. This allows us to cleanup dependency resolution and move various versions from across the project to a single file.

Solution

This PR introduces version catalogs and removes the palantir consistent versions plugin, since it is no longer needed. Additionally, the build module(s) are restructured and further changes from https://github.com/apache/lucene/pull/13484 were applied to reflect the same behavior.

Tests

Minor adjustments have been made to some test files that fix some references (due to restructuring of build files), and besides that spotless and tidy are added to run on build files too.

Open TODOs

[X] Resolve dependency resolution conflicts
[X] Update documentation about dependencies
- [X] Include new notes for reviewing license and notice files (see discussion thread)
[X] Update license and notice files of updated dependencies
[ ] Reapply forbidden APIs' signature logic in forbidden-apis.gradle (depends on https://github.com/policeman-tools/forbidden-apis/pull/249)
[ ] Review dependency bot for breaking functionality
[ ] Update Changelog file
[ ] Verify the validity of the steps for "Update Lucene prerelease" (ASF account needed)

Checklist

Please review the following and check all that apply:

[X] I have reviewed the guidelines for How to Contribute and my code conforms to the standards described there to the best of my ability.
[X] I have created a Jira issue and added the issue ID to my pull request title.
[X] I have given Solr maintainers access to contribute to my PR branch. (optional but recommended)
[X] I have developed this patch against the main branch.
[x] I have run ./gradlew check.
[ ] I have added tests for my changes.
[ ] I have added documentation for the Reference Guide

Dependecy Updates and Related PRs

This PR updates multiple dependencies (minor and patch versions) to the most up-to-date versions. These updates do not include breaking changes and therefore are applied in this PR. This should simplify conflict resolution with versions.props for any upcoming changes (see below PRs). The corresponding license and notice files for each dependency that was changed have been updated as well. Some license and notice files of libraries from the same project were merged.

In order to run a successful gradlew check (with Java 17 for now) some of the dependency issues (see #2643) had to be fixed as well.

Major upgrades or upgrades that introduce breaking changes are not included and should be addressed in different PRs.

Resolves #2047 Resolves #2129 Resolves #2130 Resolves #2167 Resolves #2169 Resolves #2268 Resolves #2396 Resolves #2398 Resolves #2608 Resolves #2609 Resolves #2628 Resolves #2637 Resolves #2639 Resolves #2638 Resolves #2643 Resolves #2647

Aug 14 '24 12:08 malliaridis

If this change is intended for Solr 10, it should be tested and developed using Java 21 to ensure there are no hidden surprises. I have one PR that makes Solr compatible with Java 21, but we later decided to break it into parts to reduce the burden for the final PR. I will prepare it by cherry-picking all the latest changes I made So that we can test this change.

Aug 14 '24 14:08 iamsanjay

@iamsanjay That would be great. We probably have to add the foojay-resolver-plugin too at some point.

Aug 14 '24 14:08 malliaridis

I think it was a mistake adding f402a0b onwards in the same PR, even thought there were only minor or patch version updates. It turned way larger than expected. Even though, over 11K out of 13K lines of code are from the versions.lock file, so maybe it is not that bad?

Maybe splitting this PR before finalizing is better.

Aug 18 '24 02:08 malliaridis

I would keep any upgrades from infrastructure changes separate. You can then diff/ compare the final artifacts and see what's changed before doing any updates.

Aug 23 '24 12:08 dweiss

Yeah I agree. I already started the process of splitting the PR into two, one for infra and one for the dependency updates. I am facing a few version differences with non-conflicting dependencies that belong to the same group (e.g. grpc dependencies).

I may have to introduce new BOMs if available to properly sync the versions across all artifacts of the same group.

Aug 23 '24 13:08 malliaridis

This PR is closed in favor to #2706, which does not include all the dependency updates made here. They will be resolved in separate PRs.

Sep 12 '24 10:09 malliaridis