feat: 🚧 WIP: Logger instrumentation

[kaylareopelle]

Description

This is a work-in-progress implementation of an OpenTelemetry logs bridge for Ruby's built-in Logger library.

It also includes patches to ActiveSupport::Logger.broadcast and the ActiveSupport::BroadcastLogger to emit only one log record for a broadcast.

It relies on a WIP Logs SDK and OTLP exporter implementation. If you'd like to test out this experimental code, follow the instructions on this gist, or clone this demo repo.

The SDK and OTLP exporter code is being reviewed in small chunks on the open-telemetry/opentelemetry-ruby repository. Follow the progress on the Logs project board.

Once the SDK/exporter code is merged into the main repo, this PR can be taken out of draft mode.

@khushijain21 is a co-author of this PR and contributed functionality as part of her LFX mentorship with OpenTelemetry in 2024.

TODOs:

• [x] Improve README
• [x] Add example
• [x] Add workaround to prevent duplicate logs in Rails 7.1
• [X] ~~Verify OpenTelemetry logs aren't emitted as LogRecords (or should they be?)~~ Seems they can
• [x] Determine whether attributes should be able to be passed to the bridge and add this functionality if they should (I'm leaning toward no because this is a bridge of an existing framework that did not have support for attributes, so it doesn't seem likely that attributes have a place to be reasonably added. We could include additional information as attributes, maybe like progname, but I don't know if that's valuable to users)
• [x] Verify where this bridge should live (currently under the instrumentation directory, but would a bridges directory be more appropriate? => Python puts logs instrumentation in instrumentation, JS put winston in packages, Go uses bridges, Java keeps it in instrumentation)
• [ ] Depending on when this is released, add support for Rails 8 structured logging

Closes #668

[github-actions[bot]]

👋 This pull request has been marked as stale because it has been open with no activity. You can: comment on the issue or remove the stale label to hold stale off for a while, add the keep label to hold stale off permanently, or do nothing. If you do nothing this pull request will be closed eventually by the stale bot

[kaylareopelle]

I'm still trying to figure out why one of the tests is failing on the CI, but passes locally. However, everything except the ActiveSupportLogger code should be ready for review. Please take a look and I'll push the update as soon as I can!

[tomash]

I am very interested in getting this to work with Opentelemetry. Any way I could help with this work?

[kaylareopelle]

Hi @tomash! Thanks for reaching out. It's great to hear there's interest in this PR. I'd love your help!

The features for the bridge should be complete. I just updated the branch to align with some of the changes in main. Before this is merged, we need to agree on the approach and get more feedback overall.

Specifically, I could use help with:

1. Reviewing the code: What are your thoughts on how OpenTelemetry logs are bridged from the Ruby Logger? Do you have suggestions on how we could improve the approach? An alternative approach has been presented in https://github.com/open-telemetry/opentelemetry-ruby/discussions/1789, would you rather see something like this?

2. Testing the code in an app: Install the gem from this branch and provide feedback on how it works in your application. This is a Gemfile from a sample application I wrote to add the OpenTelemetry bridge for the Ruby logger to a Rails application. Give it a try and let us know how it goes.

[tomash]

@kaylareopelle

For now I have this monstrosity in Gemfile:

gem "opentelemetry-instrumentation-logger",
  git: "https://github.com/kaylareopelle/opentelemetry-ruby-contrib.git",
  branch: "logger-instrumentation",
  glob: "instrumentation/logger/opentelemetry-instrumentation-logger.gemspec"

And I can confirm it works well for our app (the logs appear in Honeycomb just as events). The only problem is that Rails logs are noisy by default and it makes "top messages" unusable (top message being "rendered profiles/_carousel.html.erb"), so I've also dropped in Lograge to produce fewer logs.

I haven't seen https://github.com/open-telemetry/opentelemetry-ruby/discussions/1789 , will evaluate and compare! I do like how lightweight and easy to read it is.

[tomash]

So I'm after a few weeks of running this PR in production for a medium-sized app with a lot of traffic, together with Lograge. Works beautifully.

I've also evaluated https://github.com/open-telemetry/opentelemetry-ruby/discussions/1789 but we have a specific setup -- using both Newrelic and Honeycomb -- and that one-screener did not produce logs to both outputs (so maybe it was intercepting too hard). It also does not use OTEL_EXPORTER_OTLP_HEADERS so this PR is our choice.

The only flaw I found in this one is a very specific case.

1. Due to our traffic we do head-based sampling with OTEL_TRACES_SAMPLER=traceidratio and OTEL_TRACES_SAMPLER_ARG=0.01 which works great for sampling our traces.
2. Of course we do not want log sampling and it's all good, 100% of logs are sent out with this PR.
3. BUT because of that sampling, to have proper counts and reflect trace representation, we have OTEL_RESOURCE_ATTRIBUTES="SampleRate=100" (a reverse of 0.01 sampling fraction).
4. This SampleRate attribute seems to be also attached to log events because for each log entry Honeycomb multiplies its occurence count by 100.

[fguillen]

Hello @tomash I also would like to test this in my Rails project. Can you share your integration? Especially interested in the lograge integration. Thanks!

UPDATE: forget it, I see it just work out of the box after installing the gem with your gem call piece of art and requiring the gem.

I am sharing here my configurations:

# Gemfile
gem "opentelemetry-exporter-otlp-logs"
gem "opentelemetry-exporter-otlp-metrics"
gem "opentelemetry-exporter-otlp"
gem "opentelemetry-instrumentation-all"
gem "opentelemetry-sdk"
gem "opentelemetry-instrumentation-logger",
  git: "https://github.com/kaylareopelle/opentelemetry-ruby-contrib.git",
  branch: "logger-instrumentation",
  glob: "instrumentation/logger/opentelemetry-instrumentation-logger.gemspec"

# config/initializers/lograge.rb
Rails.application.configure do
  config.lograge.enabled = true
  config.lograge.formatter = Lograge::Formatters::Json.new

  # User id and host (this is very personal for my project)
  config.lograge.custom_payload do |controller|
    user_id = nil
    user_id = controller.send(:current_front_user).try(:uuid) if controller.respond_to?(:current_front_user, true)
    user_id ||= controller.send(:current_tester_user).try(:uuid) if controller.respond_to?(:current_tester_user, true)
    user_id ||= controller.send(:current_admin_user).try(:uuid) if controller.respond_to?(:current_admin_user, true)
    {
      host: controller.request.host,
      user_id:
    }
  end

  config.lograge.custom_options = lambda do |event|
    context = OpenTelemetry::Trace.current_span.context
    {
      trace_id: context.valid? ? context.hex_trace_id : nil,
      span_id: context.valid? ? context.hex_span_id : nil,
      user_id: event.payload[:user_id], # if available
      host: event.payload[:host],
      remote_ip: event.payload[:remote_ip],
      params: event.payload[:params].except("controller", "action")
    }
  end
end

# config/initializers/opentelemetry.rb
ENV["OTEL_TRACES_EXPORTER"] ||= "otlp"
ENV["OTEL_METRICS_EXPORTER"] ||= "otlp"
ENV["OTEL_LOGS_EXPORTER"] ||= "otlp"
ENV["OTEL_EXPORTER_OTLP_ENDPOINT"] ||= "https://myendpoint:4318"
ENV["OTEL_LOG_LEVEL"] ||= "info"

# Basic auth (I haven't test this yet)
username = "MyUser"
password = "MyPass"
basic_auth_hash = Base64.strict_encode64("#{username}:#{password}")
basic_auth_header = "Authorization=Basic #{basic_auth_hash}"s
ENV["OTEL_EXPORTER_OTLP_HEADERS"] = basic_auth_header

require "opentelemetry-metrics-sdk"
require "opentelemetry-logs-sdk"
require "opentelemetry/sdk"
require "opentelemetry/exporter/otlp"
require "opentelemetry/instrumentation/all"
require "opentelemetry-instrumentation-logger"

OpenTelemetry::SDK.configure do |c|
  c.service_name = "app.playcocola.com"
  c.use_all() # enables all instrumentation!
end

UPDATE 2: I see the author or the PR has a proper Rails demo published

[fguillen]

Funny behaviour. If I have activated ENV["OTEL_LOG_LEVEL"] ="debug". When a log is exported another DEBUG log is generated like:

D, [2025-05-01T22:42:17.912579 #95172] DEBUG -- : Successfully exported 1 log records

And it gets into an infinite loop. It may be expected. But just for you to take into consideration :)

[tomash]

@fguillen I was about to write an answer but you figured it all out in the meantime!

Sharing my slightly-different lograge.rb initializer:

Rails.application.configure do
  config.lograge.enabled = true

  # the hotwire connect-disconnect logs are just noise
  config.lograge.ignore_actions = [
    "Turbo::StreamsChannel#subscribe",
    "Turbo::StreamsChannel#unsubscribe",
    "Hotwire::Livereload::ReloadChannel#subscribe",
    "Hotwire::Livereload::ReloadChannel#unsubscribe",
    "ApplicationCable::Connection#connect",
    "ApplicationCable::Connection#disconnect",
    "ApplicationCable::Connection#reconnect",
    "ActionCable::Connection::Base#connect",
    "ActionCable::Connection::Base#disconnect",
  ]
  config.lograge.custom_payload do |controller|
    {
      params: controller.request.filtered_parameters,
    }
  end
end

In opentelemetry.rb initializer we also have this line, as I saw it in this PR's example code:

  at_exit do
    OpenTelemetry.logger_provider.shutdown
  end

Why do you use OTEL_LOG_LEVEL instead of application-global log level?

[fguillen]

hi @tomash:

Why do you use OTEL_LOG_LEVEL instead of application-global log level?

As explained in a previous comment, I can not set OTEL_LOG_LEVEL=debug and also activate OTEL_LOGS_EXPORTER. Because the system enters an infinite loop of noticing the debug log that has been sent, and by this, generating another debug log, and so on.

And regarding:

at_exit do
    OpenTelemetry.logger_provider.shutdown
end

I haven't seen it is done in the Rails demo project. So I am not doing it. Don't know if it is necessary :/

[fguillen]

I am having an issue when loading the logs in Grafana (Rails -> OtelCollector -> Loki -> Grafana). I don't know if it is related to this logger plugin or by my configuration or it is the expected behaviour.

The case is that in Grafana I see the logs like this:

{
  "body": "{\"method\":\"GET\",\"path\":\"/admin/invitations\",\"format\":\"html\",\"controller\":\"Admin::InvitationsController\",\"action\":\"index\",\"status\":200,\"allocations\":847384,\"duration\":470.19,\"view\":351.69,\"db\":86.12,\"trace_id\":\"7d5334a61dc66894a9c6ef508529de6e\",\"span_id\":\"c02fc38d3c0b7580\",\"user_id\":\"257b4cba-d430-4db7-9406-fdd08c495571\",\"host\":\"localhost\",\"remote_ip\":null,\"params\":{},\"environment\":\"development\",\"request_id\":null}",
  "flags": 1,
  "instrumentation_scope": {
    "name": "opentelemetry-instrumentation-logger",
    "version": "0.1.0"
  },
  "resources": {
    "process.command": "bin/rails",
    "process.pid": 81016,
    "process.runtime.description": "ruby 3.2.4 (2024-04-23 revision af471c0e01) [arm64-darwin24]",
    "process.runtime.name": "ruby",
    "process.runtime.version": "3.2.4",
    "service.name": "app.playcocola.com",
    "telemetry.sdk.language": "ruby",
    "telemetry.sdk.name": "opentelemetry",
    "telemetry.sdk.version": "1.8.0"
  },
  "severity": "INFO",
  "spanid": "c02fc38d3c0b7580",
  "traceid": "7d5334a61dc66894a9c6ef508529de6e"
}

The message has been created in a proper JSON format, but when it arrives at Grafana, it is in the body value and it is stringified, and I can not do proper queries on it.

I have tried many different lograge configurations but have had no luck. The body is always stringified.

[kaylareopelle]

SIG discussion 06/22/25 - Move this out of instrumentation, into a separate bridges package to make it easier to disable if people don't want to use it.

[kaylareopelle]

@xuan-cao-swi - I've looked into move the logger instrumentation into a separate bridge directory and have some concerns. I'd like to get your feedback.

1. If we move out of the instrumentation directory, should we still allow this instrumentation to depend on the opentelemetry-instrumentation-base gem? Currently, the logger instrumentation leverages the install/present/compatible methods provided by this gem. Eventually, we may want to add configuration options too, which would also use these APIs. Right now, nothing outside the instrumentation directory depends on the base gem.

2. If we move the logger logic into bridges, should we have a separate registry for bridges? Personally, I'd prefer to have one registry that holds all the possible prepended modules.

3. We'll eventually need to include creating a logger into instrumentation-base for AI events since it seems like they will be logs with the event_name attribute. This also reveals that an unnecessary tracer is being instantiated for the logger instrumentation because it depends on the instrumentation-base gem, but we could update the internals to make creating a tracer conditional.

At the moment, I'd like to keep the logger instrumentation out of all at least until the logs implementation is stable, which would add some barriers to installing it accidentally.

[wsmoak]

As discussed in... https://cloud-native.slack.com/archives/C01NWKKMKMY/p1759439218858049

I'm interested in trying this out... if you have time could you please merge in the latest changes?

[ericmustin]

lgtm once the branch conflicts r resolved

[kaylareopelle]

@ericmustin @robbkidd @xuan-cao-swi -- Made the updates we discussed in the SIG today. The logger instrumentation is no longer included in opentelemetry-instrumentation-all. Also made a few tweaks for alignment with the rest of the repo. Please take a look when you can!

[kaylareopelle]

Also, the markdown link checker will fail until this PR is merged, because there's a link related to content in the PR. https://github.com/kaylareopelle/opentelemetry-ruby-contrib/actions/runs/18325633495/job/52189286587#step:3:262

[ericmustin]

Sorry I missed this, lgtm (in hindsight!)