Structured Logging Trace Propagation for Google Cloud Logging

[thomasrichner-oviva]

Is your feature request related to a problem? Please describe.

Structured logging with the JbossLogManager does not propagate the spanId and trace in a form where Google Cloud logging can work with it. The jboss-logmanager-mdc extension can add spanId etc. to the MDC context, but this does not end up on the root object when logged and also doesn't match the keys that Cloud Logging expects.

Describe the solution you'd like

Ideally there would be instrumentation of JbossLogManager and other LogManagers that add the spanId etc. when structured logging is used. Specifically those three fields semantically understood by Cloud Logging:

• logging.googleapis.com/spanId
• logging.googleapis.com/trace (GCP specific)
• logging.googleapis.com/trace_sampled

Currently the Java LogManager is pretty limited in terms of what root level keys are allowed, so probably one would have to instrument the actual formatter. For example here.

Describe alternatives you've considered In the case of Quarkus, one could also use manual instrumentation instead of the agent, though that is less convenient.

Additional context

• Cloud Logging Documentation
• Jboss LogManager
• Exisiting MDC instrumentation

[mateuszrzeszutek]

Hey @thomasrichner-oviva ,

All of the MDC/ThreadContext instrumentations use trace_id/span_id because that how the OTel spec defines the field names for compatibility with legacy logging systems.

Also, as the scope of this repo does not include vendor-specific code, we won't change the implementation here. I'll move your issue over to the contrib repo though -- perhaps someone willing to maintain an alternative implementation will contribute it.

[thomasrichner-oviva]

Hey @mateuszrzeszutek,

Thanks for the quick response! Indeed I found now also the relevant section in the spec.

Assuming I would want to contribute a respective extension for the agent, would it be useful to have it in this repo or is that not something that fits here?

[trask]

hi @thomasrichner-oviva!

this does not end up on the root object when logged and also doesn't match the keys that Cloud Logging expects

is there a way to customize the pattern layout to emit the data in the way GCP expects? e.g. https://docs.jboss.org/process-guide/en/html/logging.html

[thomasrichner-oviva]

Hi @trask ,

Not that I found. The structured logging in Java going via the java.util.logging (JUL) facade is rather in-flexible in terms of root object properties. I think its a bit more flexible for unstructured logs, but that just brings new problems.

One idea is to write an agent extension and directly hook into the actual formatter/writer as linked above.