SOLR-17277: Add 'warnOnly' mode to CircuitBreakers

[gerlowskija]

Tripped soft (i.e. 'warnOnly=true') breakers are logged in a single message for each request, whether or not there are any 'hard' breakers that will ultimately block the request from proceeding.

https://issues.apache.org/jira/browse/SOLR-17277

Description

The threshold values used when configuring Circuit Breakers are incredibly important. Identifying when load begins to get "dangerous" is tricky: too stringent and users will get 429s unnecessarily, too lax and the circuit breakers won't be effective at preventing crashes.

It'd be much easier to find and tune these threshold values if Circuit Breakers could be enabled in a "warn only" mode, where tripped breakers would be logged but not actually "short-circuit" requests.

Solution

This PR introduces "warnOnly" mode, which can be configured on a breaker-by-breaker basis.

Circuit breakers are "hard" by default, but may be put into warn-only mode by:

• adding a <bool name="warnOnly">true</bool> setting under the <circuitBreaker/> tag for collection-level CB's.
• adding a 'warnonly'-suffixed envvar/sysprop for "global" CB's. (e.g. SOLR_CIRCUITBREAKER_QUERY_MEM_WARNONLY=true)

Tests

New test cases in TestCircuitBreakers for "warnOnly" mode and to close coverage gaps in CB configuration.

Checklist

Please review the following and check all that apply:

• [x] I have reviewed the guidelines for How to Contribute and my code conforms to the standards described there to the best of my ability.
• [x] I have created a Jira issue and added the issue ID to my pull request title.
• [x] I have given Solr maintainers access to contribute to my PR branch. (optional but recommended)
• [x] I have developed this patch against the main branch.
• [ ] I have run ./gradlew check.
• [x] I have added tests for my changes.
• [ ] I have added documentation for the Reference Guide