OpenTelemetry Middleware

[autodidaddict]

Using the middleware functionality available in the host, we should provide a crate that contains an OpenTracing middleware implementation. This should perform, at a minimum, the following:

• Initiate a new span when appropriate:
  • When a capability invokes an actor without an existing span
  • When the host performs a manual invocation on an actor (which will not have an existing span)
• Maintain the span so that a single span contains the capability->actor invocation and all subsequent actor->capability calls that result from that.
• Add functionality to the actor guest SDK to add "tracing baggage" in a vendor-agnostic way (e.g. ctx.tracing().set_baggage("key", "value"); where the actor developer does not have to maintain correlation IDs. Actors/actor SDK should not have the ability to create sub-spans or new spans.
• Add functionality to the actor guest SDK to allow a developer to manually set the span to error/failed (e.g. ctx.tracing().set_failed())
• Modify a span when an actor or capability call fails to change the span to failed/error
• Emit the appropriate UDP messages to a Jaeger collector (I don't particularly mind if the first implementation of this middleware is Jaeger-specific as it's the de-facto OpenTracing standard for CNCF)
• if there's no tracing middleware in use by the host, then the relevant actor functions will appear to have no effect. In other words, the existence or non-existence of tracing middleware should never require someone to recompile an actor.

This feature will require modifying the waSCC dispatch and multiplexing systems so that open tracing spans can be maintained across invocations.

This code should be an independently versioned repo/crate and should not be inside the main wascc-host repository.

[autodidaddict]

Per conversation with Tey, it's worth looking into the possibility of using Open Telemetry as a superset of open tracing.

[Spora-GH]

There is an official comprehensive collector which provides a lot of functionality, which makes things very simple. It contains all the receivers, exporters and processors that can be easily configured via yaml.

This collector model makes aggregation, chaining and processing telemetry across the lattice very flexible... Each collector can receive and export in any format, each receiver has its own endpoint.

This model means that the telemetry layer can remain completely separate from the wascc lattice, which will just forward traces and metrics to the receivers. When logs are integrated later on, it should be just as simple to configure.

[Spora-GH]

https://opentelemetry.io/docs/collector/about/