Improve the logs integration with `tracing`

[thomaseizinger]

Description

We make use of structured logging a lot using tracing, i.e. a lot of context of our log messages is captured in fields set either on the message or via a span. By default, the rendering of the message in the logs overview is therefore pretty useless because one needs to unfold each message to see the variables.

For our stderr logs, we use tracing_subscriber::fmt so our heads are used to parsing through logs in that format. To mimic this, we added a before_log hook that renders certain fields as key=value into the message. The code for that is here: https://github.com/firezone/firezone/blob/ac34635db891f1a57ea36e8754894f1cf4984143/rust/telemetry/src/lib.rs#L296-L321

I am not sure what the best path forward here is for improving this feature from the Sentry side. I just wanted to share this as feedback how we are using it :)

[lcian]

Hey @thomaseizinger, thanks a lot for the feedback! On one hand it's easy to set up a before_log like you did, on the other hand I imagine a lot of people could benefit from this if they're used to seeing the logs in the format outputted by fmt. We could offer an option on the layer to format the message including the tracing event fields.

[thomaseizinger]

We could offer an option on the layer to format the message including the tracing event fields.

I think the important part here would be to do it in addition to also having the structured fields for filtering.

I'd also suggest that it should be default but perhaps allow turning it off.

[lcian]

You can already filter on the tracing fields, like this

If the field name contains . or : you'll have to quote it when inputting it in the search bar (this is something we should improve):

Is this what you were looking for or something else?

[lcian]

Ah, I think maybe what you mean is more of a free text search where you can just input a string without any special operators and it will filter the logs that have that string not only in the message but also as any of the attribute names/values, right? That's also an improvement we're planning as several people asked for it

[thomaseizinger]

I simply meant that if the fmt-formatting is added, the fields should still be sent, like they are today. i.e. fmt-formatting should happen in addition to the fields, not instead of.

[lcian]

Ah got it. Yeah for sure, we would just add them to the message but not remove them from the attributes 👍