Investigate CI incorrectly passing code with compilation errors

[mbaxter]

Description

For context, see the PR comment here. In summary, CI has been passing, a change is pushed up, and it starts failing on some unrelated code. The failing code was pushed in a previous PR, and there is a sequence of passing CI checks before it finally fails here due to the MethodInputParametersMustBeFinal errorprone check.

I tried running the failing gradle task locally at the git commit where it finally failed in CI. When running locally, the task passed without failure. It looks like the -WError flag should always be applied (see build.gradle) so not sure why this is passing locally.

We should investigate this CI flakiness and make sure that we're actually enforcing all of the rules in CI consistently.

[non-fungible-nelson]

@macfarla for visibility

[usmansaleem]

@non-fungible-nelson I've picked up this issue. This resulted in taking out Besu's custom errorprone into its own repo. Also, the class involved, looks like our errorprone check is not always able to pick it, so I've make a small refactoring in it. Will create a PR to cover this and other bits that are uncovered due to upgrading the errorprone. See https://github.com/hyperledger/besu/pull/6692 as well.

[usmansaleem]

While investing this issue, the Besu's custom errorprone checks which were part of Besu codebase and were built and applied in each build task, has been moved to its own repository and now are referenced as an external dependency.

The -Werror by gradle does pass to compiler and to each submodule which results in errorprone warnings to be treated as errors during compilation. The most likely cause errorprone fails to apply to various gradle submodules when attempting to build locally is due to caching and incremental compilation "feature" of submodules. The recent changes in GH CI specially around building and running tests results in building various individual modules which causes errorprone checks to be correct applied during pre-compilation phase. As a result of this, errorprone suggestions has been applied in various submodules so that in future, GH actions should pick any errorprone violations.

Closing this issue for now.