logstash First idea in adding histogram metric

Release notes

[rn:skip]

What does this PR do?

Create a new histogram metric type. Updated all metric infrastructure code, the core class is HistogramMetric which uses HdrHistogram Recorder to track the measurements and produces a new HistogramSnapshot every time getValue is invoked, clearing the measurements stored in the Recorder. HistogramSnapshot is a data class that exposes 75Percentile and 90Percentile (at the moment).

To verify the effectiveness of this, the memory queue read client was updated to expose a metric to track the batch size.

Created a new setting named pipeline.batch.metrics with values "true" and "false" (at the moment), to enable and disable the computation of such metrics. This setting is a string because in a follow up PR will become a tri-state flag.

Why is it important/What is the impact to the user?

This is an intermediate step, it has to proof the exposition of new metric section under the _node/stats API endpoint like:

"pipelines": {
    "main": {
      ...
      "batch": {
        "event_size": {
          "p75": 12.4
          "p90": 13.1
        }
    }
  }
}

Percentiles are also pushed down to ES when leveraging the monitoring.

Checklist

[x] My code follows the style guidelines of this project
[x] I have commented my code, particularly in hard-to-understand areas
~~[ ] I have made corresponding changes to the documentation~~
~~[ ] I have made corresponding change to the default configuration files (and/or docker env variables)~~
[x] I have added tests that prove my fix is effective or that my feature works

Author's Checklist

[x] Check with xpack monitoring

How to test this PR locally

Run Logstash with the setting pipeline.batch.metrics with values "true" and verify that node stats exposes the batch_size histogram.

curl http://localhost:9600/_node/stats | jq .pipelines.main.events

Related issues

Relates #17895
Closes #17973

Use cases

Screenshots

Logs

Aug 01 '25 08:08 andsel

:robot: GitHub comments

Expand to view the GitHub comments

Just comment with:

run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

Aug 01 '25 08:08 github-actions[bot]

This pull request does not have a backport label. Could you fix it @andsel? 🙏 To fixup this pull request, you need to add the backport labels for the needed branches, such as:

backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit.
If no backport is necessary, please add the backport-skip label