index setting at module level configuration ignores output.elasticsearch.pipeline in Metricbeat

[eedugon]

When using the index setting at module level documented here, the ouput.elasticsearch.pipeline setting is not respected.

Configuration example:

cloud.id: "test-8131:REDACTED"
cloud.auth: "elastic:REDACTED"

output.elasticsearch:
  hosts: ["localhost:9200"]
  pipeline: amend-system-data

- module: system
  period: 10s
  index: metricbeat-override
  metricsets:
    - cpu
    - load
    - memory
    - network
    - process
    - process_summary
    - socket_summary

  process.include_top_n:
    by_cpu: 5      # include top 5 processes by CPU
    by_memory: 5   # include top 5 processes by memory

- module: system
  period: 1m
  index: metricbeat-override
  metricsets:
    - filesystem
    - fsstat
  processors:
  - drop_event.when.regexp:
      system.filesystem.mount_point: '^/(sys|cgroup|proc|dev|etc|host|lib|snap)($|/)'

- module: system
  period: 15m
  index: metricbeat-override
  metricsets:
    - uptime

Pipeline:

GET _ingest/pipeline/amend-system-data
{
  "amend-system-data": {
    "processors": [
      {
        "set": {
          "field": "test",
          "value": "value"
        }
      }
    ]
  }
}

Result: the field that the simple amend pipeline is adding, is not created, as the pipeline is not even called.

Expectation: Users would expect the pipeline to be respected as it's configured in the output and the system module is not using its own pipeline.

Based on a conversation with @lucabelluccini and @MichaelKatsoulis we probably prefer to not change the current behavior and just clarify better the docs the current behavior to avoid incorrect expectations.

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[botelastic[bot]]

Hi! We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!