[MVP] Tooling for benchmarking Elasticsearch ingest changes

[simitt]

Problem

We would like to increase our visibility into the effects of changing Elasticsearch index mappings, and ingest pipelines. For example, what would be the change in indexing time and Elasticsearch storage size if we were to start mapping SQL statements with the text field type?

By introducing tooling to analyse these kinds of changes, we can make changes more quickly, and with confidence.

Expected Outcome

Build an MVP so that developers can run benchmarks on-demand, from their laptop, to analyse the impact of integration packages on Elasticsearch performance and storage cost. The output should be a Rally summary report.

• [ ] Ability to inject a custom apm integration package into a Kibana image
• [ ] Spin up ESS deployment of a specific version, allowing to override the Kibana image to use the one with the custom apm integration package.
• [ ] Generate the ES corpus from APM agent data sent to an APM Server of a specific version (locally or directly on worker VM).
• [ ] Run rally on the worker VM to benchmark the Elastic Cloud cluster

Outside of scope

• benchmarking query performance
• indexing results in Elasticsearch
• no CI integration for running benchmarks regularly

Relates to

https://github.com/elastic/apm-server/issues/6115

[simitt]

Ability to inject a custom apm integration package into a Kibana image

https://github.com/elastic/kibana/issues/137759 seems relevant for this

[lahsivjar]

#8856 Introduces support for injecting a custom APM Package into kibana image as well as ability to spin-up ESS deployment with a custom integrations package.