opentelemetry-rust icon indicating copy to clipboard operation
opentelemetry-rust copied to clipboard

OTel Logs with no heap allocation

Open cijothomas opened this issue 7 months ago • 0 comments

Opening a parent issue to track making it possible to do OTel Logging with zero heap allocation. This requires support from

  1. Logging library (tracing is used as example, which already has zero allocation and delegates most work to layer/subscriber)
  2. OTel Bridge API - this delegates most work to OTel Logging SDK and has no heap allocation now.
  3. OTel Log SDK - Currently, this is where all heap allocation occurs!
  4. Exporter - Exporting to etw, user_events can be synchronously, and hence can be fully done without heap allocation.

Making OTel Log SDK heap allocation free requires addressing the below at the minimum:

  1. https://github.com/open-telemetry/opentelemetry-rust/issues/1919 - Avoid vec! for passing LogRecord to Exporter.
  2. https://github.com/open-telemetry/opentelemetry-rust/issues/1920 - Avoid allocating vec! for storing upto "N" attributes
  3. https://github.com/open-telemetry/opentelemetry-rust/issues/1921 - Avoid String allocation. For tracing mostly

For sync exporters

The overall flow would be (using tracing as example) tracing -> otel-tracing-layer -> create LogRecord on stack, populate it using callback/visitor from tracing-subscriber. -> pass mut ref of LogRecord to exporting processor -> exporter ->serialize and write to transport (etw/user-events/lttng)

The above can be achieved fully on stack. This issue is targeting the above only.

For async exporters For exporters requiring Batching for efficiency (eg: OTLP over gRPC, HTTP), some Ring buffer or Channels would be needed. There would be clone/copy costs for those scenarios, but it maybe possible to avoid heap allocation entirely. This will be tracked separately.

Also need some tooling to ensure the assumptions are correct. i.e measure the actual heap allocation and confirm it is indeed 0! Above is based on manual code inspection only.

cijothomas avatar Jul 23 '24 01:07 cijothomas