opensearch-project
[BUG] Exceptions in AbstractSearchAsyncAction.innerOnResponse() are only logged with TRACE level
Describe the bug
Errors in onShardResult() in AbstractSearchAsyncAction.innerOnResponse() are caught and logged with only trace level, with the description "Failed to consume the shard {} result: {}" (link. This can happen for example if there's any error while serializing shard-level aggregation results, such as InternalMatrixStats, which makes debugging tricky if you don't know what exception is happening.
From the user perspective, the failed aggregation then just hangs, and if the OpenSearch logs aren't showing TRACE level, which is very likely, there's no indication what went wrong, or if the aggregation is stuck processing.
Related component
Other
To Reproduce
Introduce some error in an aggregation serialization code, for example at InternalMatrixStats.doWriteTo(). Run that aggregation. See it appear to hang from the client side.
Expected behavior
It would be better if the exception was surfaced to the user in a way they'll actually see. I think this should be WARN level, or if this class is also used in places where we expect exceptions to be frequent, we should put logging higher up in the stacktrace to capture issues with specifically serializing aggregation results.
Here's my example IntelliJ debugger stacktrace for reference:
Additional Details
I can see that most of the failures in AbstractSearchAsyncAction are reported as debug, which also does not seem correct. While intuitively this sounds like the right thing, I want to be cautious while logging these exceptions as WARN and ensuring that the logs are not flooded with these errors/warnings
