lunatic icon indicating copy to clipboard operation
lunatic copied to clipboard

feat: add opentelemetry tracing and metrics

Open tqwewe opened this issue 2 years ago • 1 comments

Todo:

  • [x] Each process should have its own single context.
  • [x] Ensure doc comments are written correctly.
  • [x] Make sure host function trait bounds are only whats required.
  • [x] Figure out how to add the environment id to prometheus target_info. (Turns out its not possible)
  • [x] ~Add push and take functions for resource sharing. Though, this should not "move" the resource, probably would be better to allow multiple processes to share metric resources.~ I don't think this is needed in this PR for now.
  • [ ] What do I do with distributed state?

Spans cannot be shared across processes, as they are in a tree structure, and sharing them means it would be possible to drop a parent span before its child, which wouldn't make sense.

Running the spawn process benchmark, this PR does not seem to affect performance of spawning processes.

Related PRs: https://github.com/open-telemetry/opentelemetry-rust/pull/1009 https://github.com/open-telemetry/opentelemetry-rust/pull/1018


Screenshots for examples/metrics.rs

https://github.com/lunatic-solutions/lunatic-rs/blob/4681561eb78d1164bc1b2eef7c436bcab36622ab/examples/metrics.rs#L21-L78

Terminal

[2023-04-06T07:22:53Z INFO  metrics] Additional log message, with formatting!
[2023-04-06T07:22:53Z INFO  metrics] formatted object
[2023-04-06T07:22:53Z INFO  metrics] debug object
[2023-04-06T07:22:53Z INFO  metrics] person object
[2023-04-06T07:22:53Z INFO  my_app] a log from my_app
[2023-04-06T07:22:53Z INFO  metrics] a log under my_span

Jaeger jaeger

Prometheus prometheus

tqwewe avatar Mar 28 '23 09:03 tqwewe

Overall it looks good! All metrics include some attributes like: node id, environment id, process id. They are not implicitly set, right? We have to set them on each call? We also want to set them in vm, no guest, so that it can be trusted.

Actually the process_id and environment_id is not attached to every trace/metric, but is only attached to the parent spans. This might be a problem though since if the runtime exits without closing a span properly (which is quite likely), then we might not know which environment the spans come from.

I'll work on injecting this data to every span/log/metric.

tqwewe avatar Apr 18 '23 05:04 tqwewe