Exception stack trace to string conversion customization

[alex-kormukhin]

Is your feature request related to a problem? Please describe. If span has exception (recordException) with few wrapped cause and long stack traces (some spring exception), the string representation in tracing system (tag exception.stacktrace) has "caused by, root cause last" structure. This is sometimes inconvenient. In log we use logback %rEx directive for "wrapped by, root cause first" stack trace structure (like this). Also in logging with "wrapped by, root cause first" we can painlessly truncate wery large stack traces - root cause does not truncated. But with "caused by, root cause last" structure it is not.

Describe the solution you'd like It would be nice to be able to customize exception stack trace to string conversion in open telemetry by user. Inject users printStackTrace implementation. Or have open telemetry settings like logback %rEx and stack trace size limit.

[jkwatson]

This would be something added to the Tracing SDK, correct? recordException is currently just semantic sugar for adding an event to the span. I think you could create your own implementation that adds that event with the precise contents that you're interested in, without a lot of work.

[alex-kormukhin]

I mean hardcoded exception.printStackTrace(printWriter) in jaeger and otel span exporter:

We use manual instrumentation, not javaagent instrumentation (spring sleuth with otel). How can we customize that?

[jkwatson]

That class you see there, "ImmutableExceptionEventData" is what is generated by the SDK when you span.recordException(...). If you don't use that method to record your exceptions, you can format the resulting event content however you like.

[alex-kormukhin]

Directly via addEvent(String name, Attributes attributes)? Ok. Thanks.

[jkwatson]

Directly via addEvent(String name, Attributes attributes)? Ok. Thanks.

I do think your issue is a legitimate one. I'm not sure what the right solution is, though. If you could prototype a concrete proposal, it would be helpful.

[alex-kormukhin]

Oh, we can't replace recordException calls to addEvent with third party instrumentation:

So customization on opentelemetry SDK level is very useful.

If you could prototype a concrete proposal, it would be helpful.

Unfortunately, there isn't much time for that right now. Maybe I can work on that a little later. Sorry :(

[jkwatson]

Yes, we are very very understaffed in otel-java at the moment. Only one full time maintainer, and me, who is part-time. So, any significant changes like this will require community help.

[philsttr]

I want to offer a possible example of how stacktraces could be customized. The logstash-logback-encoder library (of which I am a maintainer) contains a really powerful way of customizing how stacktraces appear in logs. But the same concept could apply to the exception.stacktrace attribute in otel.

Perhaps the implementation could be used as an inspiration as to what options would be beneficial in otel. For example, the implementation can:

• omit common frames between causes
• truncate the stacktrace after a frame matches a regex (e.g. where a framework calls into your code)
• exclude frames based on regex (e.g. remove proxy or aop frames)
• limit max frames per throwable
• limit max bytes per stacktrace (not really needed if you specify attribute limits in otel)
• shorten classnames (e.g. abbreviate package names)
• put root cause first

All of these options allow stacktraces to be more useful by filtering out noise.

[JnRouvignac]

I would also like to "customize exception stack trace to string conversion in open telemetry". We have come up with a more compact representation for exceptions:

Exception: message1 at MyClass.java:100,200 MyOtherClass.java:300 / (1) caused by Exception: message2 at SomeClass.java:111 SomeOtherClass.java:222,333 / (2) caused by IOException: message3 at AnotherClass.java:1111 SomeAnotherClass.java:2222,3333

Thanks @jkwatson for suggesting how to do it today.