Better Tracing

[balegas]

We have set up basic Tracing for Electric but we want to continue improving what we become more successful investigating issues.

• We're following our intuition a bit and we will continue. Having real traces that we need to investigar help us focus on the things that need more observability
• in some cases we're abusing the use of spans which migh lead to high volumes of traces (these are billed). We should try to replace some traces for attributes and reserve the finer details for a TRACE mode that we can activate to increase the level of details to investigate incidents

[balegas]

note https://github.com/electric-sql/electric/issues/1738

[balegas]

in some cases we're abusing the use of spans which migh lead to high volumes of traces (these are billed). We should try to replace some traces for ...

We want to be able to turn-on this level of detail if we need to investigate any occurrences, but during normal execution we should zoom out on some of the details and put more details into the attributes. I'm not sure if there is any pattern to do that with OTEL, otherwise we could put them behind a flag.

For the top-level number of transactions we handle, the developer can setup sampling

[robacourt]

We should try to replace some traces for attributes and reserve the finer details for a TRACE mode that we can activate to increase the level of details to investigate incidents

Beware of making the code complicated in order to achieve this!

A great way to reduce the number of traces is Sampling: https://opentelemetry.io/docs/concepts/sampling/ You can reduce traffic 10/100 fold while not losing detail.

[KyleAMathews]

@balegas more work needed here?

[balegas]

@icehaunter let's add the Electric metrics to honeycomb.