make log_sending=true correlate to the correct service

[wolframhaussig]

Is your feature request related to a problem?

This issue is based on my post on Discuss: https://discuss.elastic.co/t/room-for-improvement-for-log-sending-true/330602/1

We have a Tomcat container running with multiple applications. Each application is a Spring webapplication deployed as a War-file. we use the apm agents service-name-auto-detection to get the correct service.name for each app. this looks like this:

• tomcat-application
  • App 1
  • App 2
  • ...

When activating log_sending, the logs are transferred without errors to the Elastic Stack, but the Logs are not correctly correlated to the matching service. Instead, we see the following:

• tomcat-application
  • Logs App 1
  • Logs App 2
  • Logs ...
• App 1
  • Traces
  • Errors
• App 2
  • Traces
  • Errors
• ...

The agent seems to instrument the logback appenders and has no reference to the original application (event.dataset is set to tomcat-application.ApplicationAAppenderDev, service.name is tomcat-application). This means, calling https://my-kibana-host:5601/s/my-team/app/apm/services/ApplicationA/logs does not show the application logs.

Describe the solution you'd like

I would prefer a solution where the log_sending logic detects the correct service.name the same way as the traces are generated. So I would expect this in Kibana APM UI:

• App 1
  • Traces
  • Errors
  • Logs
• App 2
  • Traces
  • Errors
  • Logs
• ...

Describe alternatives you've considered

As a workaround we could filter based on the event.dataset which seems to be set to the service and logging-appender name. This works for correlating data manually as we have specially named appenders but this will not show the Logs correctly in the Kibana APM UI for the correct service.

Additional context

Kibana: 8.7.0 APM integration: 8.7.0 APM Java Agent: 1.36.0

[JonasKunz]

I suspect this issue also applies to log reformatting and log correlation. IINM, we don't handle multi-service deployments within the same JVM there yet.

I think the correct solution would be to discover the service name based on Thread.currentThread().getContextClassLoader() at the time the log event is generated.