Use separate traces for each E2E client connection

[mostafa]

Currently, the traces are all published under the same context, which is GatewayD, hence a single trace object with many spans across different parts of the system. The single-trace-multiple-spans approach makes it hard to debug individual connections between each client and database server. By creating a new trace with random UUID, we can easily track separate connections, without trying hard to find it inside the vast amount of spans produced by GatewayD in a single trace.

[eabasir]

@mostafa If the goal is to separate the traces of clients to better observe them, cannot we just create a trace with a new root at this line:

clientCtx, span := otel.Tracer(config.TracerName).Start(ctx, "NewClient", trace.WithNewRoot(), trace.WithNewRoot())

It must be enough to split traces of clients. Right?

[mostafa]

@eabasir This only shows half the traces, namely those that happened between GatewayD and postgres (DB), and not the ones between the client (e.g. psql) and GatewayD.

[eabasir]

@eabasir This only shows half the traces, namely those that happened between GatewayD and postgres (DB), and not the ones between the client (e.g. psql) and GatewayD.

Gotcha! Okay. I'll take this!

[mostafa]

Basically you should trace when the client connects until it disconnects and observe and compare the changes before it. This way you can see what is important to an E2E trace.

[mostafa]

I guess this can be moved back to the backlog until one has time to do it.