Support OTel Collector Pipeline to Capture Traces

[kdhamric]

Enable the collection of just Tracetest spans via the configuration of the OTel Collector. The configuration / custom processor with send Tracetest spans to the Tracetest instance, which will put them in its own datastore (Jaeger?). This will enable us to work with any customer using the Otel Collector.

[mathnogueira]

Design proposal

flowchart TD

Tracetest -- 1. Send traceID to watch --> Collector
Tracetest -- 2. Trigger transaction --> System[System under test]
System -- 3. Send spans --> Collector

Collector -- 4 --> Processor(Tracetest Collector Processor)
Processor -- 5 --> Filter{is a test span?}
Filter -- No --> Drop[Drop span]
Filter -- Yes --> Exporter(Tracetest Collector Exporter)
Exporter -- 6. Send span --> Tracetest

1. Tracetest will generate a traceID and send it to the collector so it knows it must watch for any span from that trace;
2. Tracetest triggers the service under test by sending a request to it and propagating the generated trace context
3. The service under test and its dependencies will generate its spans using the same traceID that Tracetest generated in step 1 and send them to the OTEL collector;
4. The collector will have a special pipeline that will use Tracetest's processor and exporter. It will process all spans;
5. Our processor will check the list of traceIDs that are being watched at the moment and see if any span is part of any of those traces. If a span is part of no watched trace, it will be dropped. Otherwise, it will be passed to the next stage of the pipeline;
6. The exporter will receive all spans that are part of a watched trace and it will send them to Tracetest. Inside Tracetest, each span will be saved into the right test run object using the traceID to identify it.

Notes

• This will be done in a new pipeline, so the user's existing pipeline will be untouched.
• Only transactions triggered by Tracetest will be sent to the Tracetest instance

Discovery tasks

• [ ] Figure out if it's possible for a OTEL processor to listen to a TCP port: if it is possible, Tracetest will use this port to send the traceID to be watched;

Requirements

• Sending a traceID from Tracetest to the collector must be a synchronous operation
• Tracetest needs to send a signal to the processor to make it remove a traceID after the test finished running
• The OTEL processor will have to store the list of traceIDs in an in-memory structure for fast comparisons (no external dependency)