ES|QL: add Telemetry API and track top functions

[luigidellaquila]

This adds MeterRegistry (ie. APM telemetry) to ESQL.

It also adds logic to track most used commands (as it already did with the phone-nome API) and most used functions.

The responsibility of collecting telemetry is now on EsqlSession, so that it can easily be delegated to lower level components if we want to collect further information.

The legacy Telemetry is still there, untouched

[elasticsearchmachine]

Hi @luigidellaquila, I've created a changelog YAML for you.

[elasticsearchmachine]

Pinging @elastic/es-analytical-engine (Team:Analytics)

[luigidellaquila]

Thanks for the review and for the offline chat @astefan

let's leave the "legacy" metrics classes mostly untouched and the logic that belongs to the telemetry API to be moved in a separate structure of classes

That's fine, I don't have a particular preference for keeping them together or separate; happy to split them and leave the old components untouched.

we need more tests [...] We shouldn't count the eval in metrics counters

Good point! We are collecting metrics before the optimization phases, so I'm pretty sure it works as expected, but we definitely need some test coverage for this.

[luigidellaquila]

@elasticmachine update branch

[luigidellaquila]

@elasticmachine run elasticsearch-ci/bwc-snapshots

[luigidellaquila]

@elasticmachine update branch

[astefan]

Discussed this offline with @luigidellaquila some more. Notes:

• this needs more tests with metrics (due to its special form), inlinestats (because it has the multi-phase execution) and some operators (like, rlike, in, is null, is not null).
• create a followup issue for considering the above mentioned operators as being tracked or not
• for now use commandName in LogicalPlan with the mention that this likely might change (and might not be needed at all in such a broad list of classes or even completely gone) considering work around metrics like this one. We might need a different mechanism for metrics use cases telemetry.

[luigidellaquila]

Thanks for adding the off-line notes @astefan

this needs more tests with metrics (due to its special form), inlinestats (because it has the multi-phase execution) and some operators (like, rlike, in, is null, is not null).

I'm adding some tests 👍

create a followup issue for considering the above mentioned operators as being tracked or not

https://github.com/elastic/elasticsearch/issues/112449

for now use commandName in LogicalPlan with the mention that this likely might change (and might not be needed at all in such a broad list of classes or even completely gone) considering work around metrics like https://github.com/elastic/elasticsearch/pull/112383. We might need a different mechanism for metrics use cases telemetry.

Yeah, I expect we'll have to follow the evolution of METRICS and other commands and adapt accordingly

[luigidellaquila]

Thanks for the feedback folks!