open-telemetry
Infrastructure for pipeline instrumentation
The pipeline and its components must be instrumented to provide an external observability system with an accurate representation of the system’s internal state, enabling effective monitoring, debugging, and troubleshooting.
The pipeline engine adopts a thread-per-core, share-nothing architecture to minimize synchronization primitives, optimize the usage of various cache levels, and maximize data locality.
Consequently, the instrumentation framework must:
- Be optimized for thread-per-core operation
- Minimize the use of synchronization primitives to the absolute necessary. Synchronization MUST NOT occur each time a metric is updated or a new event is generated.
- Optimize memory usage (e.g. update metrics in-place).
- Conform to Semantic Convention. Medium term, leverage Weaver to generate our code for signals emission.
Possible approach: Each component (receivers, processors, exporters) will maintain their metrics/events locally, the pipeline engine will scraped these telemetry objects to emit them on a schedule. Each pipeline engine instance will report the collected signals (e.g. every 5s) on a channel locally shared across pipelines. The consumer of this channel could leverage the existing Rust Client SDK.
Other approach to explore: If the existing Rust Client SDK compatible with the requirements mentioned above, we could used it directly from the pipeline instances.
Here is an example of metrics currently used in the Go collector: https://github.com/open-telemetry/opentelemetry-collector/blob/main/processor/batchprocessor/documentation.md
I made some changes to the issue description to clarify the requirements that matter most to me.
@jmacd, @cijothomas, @utpilla, could you please review the issue, provide feedback, and potentially add further details based on your knowledge of the client SDK? Thanks!
Other approach to explore: If the existing Rust Client SDK compatible with the requirements mentioned above, we could used it directly from the pipeline instances.
OpenTelemetry Rust SDK for Metrics does NOT follow a shared-nothing model. The hot path for recording measurements involves looking up a RwLock protected Hashmap and atomic operations to make the actual update to the metric trackers (for Histogram updates, the SDK uses a Mutex as it's not feasible to use atomic operations for it).
One alternative is to create a MeterProvider instance for each thread. That way the threads don't work on shared data. They would still acquire RwLock and use atomic operations but they wouldn't face any contention so there wouldn't be much of a performance overhead. However, this approach would require more care in aggregating the data reported by the individual threads. There is a related issue where someone tried to compare the performance of using this approach: https://github.com/open-telemetry/opentelemetry-rust/issues/1386#issuecomment-2653115603
+1 to what @utpilla said. Metrics is currently designed to have a central/shared aggregator, and is the main reason for contention. The idea of creating one meter provider per core does mitigate most of the concerns (though it'd create as many threads as core, if using PeriodicReader), and is shown here: https://github.com/open-telemetry/opentelemetry-rust/pull/2659
The idea of each component maintaining own aggregation, and using ObservableInstruments/Producer in Otel SDK could mitigate this - unless Otel Rust SDK itself can be enhanced to offer instruments that are not Send/Sync which does this automatically. It's good to discuss this in one of the SIG meetings. Improving Otel Rust Metrics SDK itself to support this has been a common ask that was never prioritized due to lack of bandwidth.
Logs don't inherently have this problem. It is truly "share-nothing", and our tests (not with OTLP Exporter, but with ETW/user_events exporter) show it nicely scaling, without any contentions. (OTLP Exporter won't scale due to the usage of BatchLog/SpanProcessor, which introduce contention, though it could be redesigned to use thread_local! buffers)
See perf results for etw and user_events.
The most important missing part of this issue is the integration with the Rust Client SDK to report signal produced by the engine over OTLP