OpenSearch
OpenSearch copied to clipboard
[BUG] Level aggregations for node stats API should be processed on data nodes
Describe the bug
While reviewing #13237, I noticed that the node stats API does not pass the value of level parameter from coordinator to data node. Taking example of indices stats, that is inefficient due to 2 reasons:
- The data/coordinator nodes are serializing/deserializing all index shards information while that is not generally needed
- The coordinator node alone is responsible for aggregating the information from all the nodes, which can increase the overall latency of this operation
Related component
Other
To Reproduce
The writeTo method is serializing the most granular information for transport layer and the filtering is happening on the rest layer
Expected behavior
The data nodes should return only the information used for preparing the client response
Additional Details
Plugins Please list all plugins currently enabled.
Screenshots If applicable, add screenshots to help explain your problem.
Host/Environment (please complete the following information):
- OS: [e.g. iOS]
- Version [e.g. 22]
Additional context Add any other context about the problem here.
Thanks @jainankitk for reporting this issue. Agreed, there are other such APIs like _cluster/stats which also queries ShardStats which can be huge for a large cluster with several 10s of thousands of shards. @Pranshu-S has started looking into aggregating these shards stats on individual data nodes itself.
@Pranshu-S - @peteralfonsi has done basic implementation in #13237 for avoiding this issue with cache stats. We can leverage something similar or come up with better way to achieve this.