config.rails_semantic_logger.filter doesn't apply to initialized Loggers

[fractaledmind]

trafficstars

Environment

Provide at least:

• Ruby Version: 3.0.1
• Rails Version: 6.1.3.2
• Semantic Logger Version: 4.8.0
• Rails Semantic Logger Version: 4.6.0
• Other Application/framework names and versions
  • puma: 5.3.2
• Rails configuration. Only need the settings related to Rails Semantic Logger and Semantic Logger.

config/application.rb

config.rails_semantic_logger.started    = true
config.rails_semantic_logger.processing = true
config.rails_semantic_logger.rendered   = true
config.log_tags = {
  request_id: :uuid,
  ip:         :remote_ip,
}
config.rails_semantic_logger.quiet_assets = true
config.rails_semantic_logger.filter = Proc.new { |log| log.message !~ /Webhooks::GithubController/ }

config/environments/production.rb

if ENV["RAILS_LOG_TO_STDOUT"].present?
  $stdout.sync = true
  config.rails_semantic_logger.add_file_appender = false
  config.semantic_logger.add_appender(
    io: $stdout,
    level: config.log_level,
    formatter: config.rails_semantic_logger.format,
    filter: Proc.new { |log| log.message !~ /Webhooks::GithubController/ }
  )
end

# The log level is usually set with the config setting config.log_level,
# but Heroku also allows the log level to be set via the LOG_LEVEL env variable.
if ENV["LOG_LEVEL"].present?
  config.log_level = ENV["LOG_LEVEL"].downcase.strip.to_sym
end

I have a Rails app running on Heroku using the Papertrail service for logs. My Rails app is integrated with a GitHub app, so I receive GitHub webhooks. I store the webhook payloads in the database for processing in a background job, and they are large and often, so they fill up my logging bucket fairly easily. I am trying to exclude the logs of the GitHub webhook requests. You can see the filter I am trying to use above, but I still see the webhook requests in my logs.

How can I exclude logs in Heroku production that match my regex?

[danielwellman]

Hello, I use Semantic Logger with Heroku and Papertrail - I don't know why the code you posted isn't working, but I can share what we do -- you may already be doing something like this.

We use the feature in Papertrail to filter out log messages matching certain regular expressions. It's in the Papertrail to menu > Settings > Account page and then the "Filter logs" button. From there you specify the regular expressions that should be omitted when matched, so you might be able to add "Webhooks::GithubController" as a rule.

Here are the docs: https://documentation.solarwinds.com/en/success_center/papertrail/content/kb/how-it-works/log-filtering.htm

From what I can see, it looks like you have the syntax correct for the filter argument to add_appender; I'm looking at the documentation comment here:

https://github.com/reidmorrison/semantic_logger/blob/3f01cbb64fb8c1bbca14401703d81287399151ec/lib/semantic_logger/semantic_logger.rb#L132-L136

[reidmorrison]

Looking at the filter above, looks to me like you want to filter on the class name, not the message.

filter: Proc.new { |log| log.name != "Webhooks::GithubController" }

The log object has many elements that can be filtered on: https://github.com/reidmorrison/semantic_logger/blob/master/lib/semantic_logger/log.rb#L13

[fractaledmind]

@reidmorrison: I have found the source of the problem. The filter that I set in my configuration is never passed to the initialised loggers when the Engine is mounted.

Consider https://github.com/reidmorrison/rails_semantic_logger/blob/c94ae1f539ec1cd024462047b3ea1a4899fa8d07/lib/rails_semantic_logger/engine.rb#L45-L71

When SemanticLogger[Rails] is called, there is no way or attempt at connecting the filter created for config.rails_semantic_logger.filter. It is only used if the config.rails_semantic_logger.add_file_appender is passed and then the filter is passed to the appender.

I currently am just brute force hacking around this with an initializer:

ObjectSpace.each_object(SemanticLogger::Logger).each do |logger|
  logger.filter = Rails.application.config.rails_semantic_logger.filter
end

[fractaledmind]

To clarify, because Heroku requires logging to STDOUT (https://logger.rocketjob.io/rails#log-to-standard-out) and thus requires disabling the file appender, currently any of the options passed and used to configure the appender are ignored