[Metrics UI] Use individualized logger in metric rule executors

[weltenwort]

:notebook: Summary

Since #142121 the rule executor receives an individualized logger instance via the executor arguments. In order to correlate log messages to the rule executions we should use that instead of more general logger instances in our alerts.

:heavy_check_mark: Acceptance criteria

• All usages of loggers in the metric-related rule executors use the logger provided via the executor arguments instead of a logger injected via a create* closure.

[elasticmachine]

Pinging @elastic/infra-monitoring-ui (Team:Infra Monitoring UI)

[jennypavlova]

I added a draft PR with the change to using the logger from the executor arguments. I got a different format of the trace logs as a result from what we had before: We had:

[TRACE][plugins.infra.metricThresholdRule] [AlertId: ID][ExecutionId: ID] Request: {"index":"metrics-*,metricbeat-*","body":{....}}...

With that change I see:

[TRACE][plugins.alerting.metrics.alert.threshold] Request: {"index": "metrics-*,metricbeat-*", "body": {.....}} ....

I am not sure if that is expected so I asked the kibana-alerting team about it. Adding my questions here (cc @elastic/response-ops ) Is that the correct format I should expect and is there any way to show the ids inside the log? In order to show the ids I tried a similar config to the one from the documentation but it didn’t work for me. Config I tried:

logging.loggers:
   - name: plugins.alerting
     level: trace

Also this one:

logging.appenders:
  console_with_meta:
    type: console
    layout:
      type: pattern
      highlight: true
      pattern: "[%date][%level][%logger] %message [%meta]"

logging.loggers:
  - name: plugins.alerting
    level: trace
    appenders: ["console_with_meta"]

My old config:

logging.loggers:
   - name: plugins.infra.metricThresholdRule
     level: trace

Side question: Do we have backward compatibility with the logger configuration - from what I checked locally if the config has plugins.infra.metricThresholdRule set and the plugins.alerting is missing there then the logs won’t be visible.

I will move the issue back to the backlog until we have an answer so we can find a possible solution.

[roshan-elastic]

Hey @weltenwort @jennypavlova - wondering if anyone could explain the value/implications on this one?

Trying to understand the benefits of this to help prioritise this...

[weltenwort]

@roshan-elastic, as part of https://github.com/elastic/kibana/issues/143211 the response ops team asks all owners of rules to migrate from custom logger instances to the framework-provided logger. The benefit of this would be that the verbosity of log messages emitted by all rules can be configured in a consistent way. I'm not sure what other changes are planned that rely on that. The meta issue currently has an 8.8 label.

[roshan-elastic]

Cheers @weltenwort - makes sense.

Would this effectively impact anywhere that we embed logs into Metrics UI?

i.e. Hosts (once we deploy the Logs tab) and inventory (where the Logs tab is in the 'fly-out' type bit)?

[weltenwort]

Would this effectively impact anywhere that we embed logs into Metrics UI?

No, it's purely an alerting rule change.

[roshan-elastic]

Hey @weltenwort, I'm very late on this.

What's the impact if we don't do this? Trying to understand how to prioritise this.

[elasticmachine]

Pinging @elastic/obs-ux-infra_services-team (Team:obs-ux-infra_services)

[botelastic[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[smith]

Is this something @elastic/obs-ux-management-team has interest in prioritizing?