Failed to apply Datadog::Contrib::Rails::Patcher patch. Cause: can't modify frozen Array

[drqCode]

Hi!

I'm using:

ruby 2.7.4 Rails 7.0.3 ddtrace 0.54.2

This is my config:

# /config/initializers/datadog.rb

require 'ddtrace'

Rails.application.config.to_prepare do
   Datadog.configure do |c|
     c.analytics_enabled = true
     c.runtime_metrics_enabled = true
 
     c.tracer tags: {
       env: Rails.env.to_s,
       revision: Revision.value
     }

     c.tracer.enabled = true
     c.profiling.enabled = true
 
     c.use :rails, service_name: 'my_rails_app'
     c.use :graphql, schemas: [Graphql::Schema], service_name: 'my_graphql'
   end
 end

I'm getting this error when starting the Rails app:

ERROR -- ddtrace: [ddtrace] (/usr/local/bundle/gems/ddtrace-0.54.2/lib/ddtrace/contrib/patcher.rb:42:in `on_patch_error') Failed to apply Datadog::Contrib::Rails::Patcher patch. Cause: can't modify frozen Array ... Location: /usr/local/bundle/gems/actionpack-7.0.3/lib/action_dispatch/middleware/stack.rb:102:in `insert

I have no choice to run Datadog configuration outside #to_prepare block since I have a big GraphQL::Schema to be autoloaded.

Any ideas how to apply ddtrace's Rails patcher before Rails middlewares stack is getting frozen?

[delner]

The technical explanation is ddtrace adds instrumentation to your Rails application by adding middleware to the Rails middleware stack. This is okay to do, however certain operations in Rails or other 3rd party libraries may "finish" or "seal" the middleware stack, effectively freezing this Array. This error suggests something caused the Rails middleware stack to be "finished" before ddtrace had a chance to add its instrumentation.

This doesn't happen on a fresh Rails 7 app. My guess would say you have something else in your application that modifies the middleware stack, or attempts to access certain things from Rails forcing Rails to "finish" the middleware stack prematurely. We won't be able to debug this without a means of reproducing the error on our side. My suggestion is to try disabling certain plugins or customizations until ddtrace appears to work again. Then that might give us a clue.

[olkeene]

It happens on Rails#7.0.2.4 only in the case when rails instrumentation was set in the way as in the initial comment or in Rails.application.config.after_initialize block, so then ddtrace tries to insert its own middleware in rails middleware frozen stack

[olkeene]

@drqCode I had the same issue since we did the setup of instruments in after_initialize block Here is how I solved that on dd-trace-rb#v1:

 require 'ddtrace/auto_instrument' # Note: This must be done after requiring any supported libraries or frameworks.
  Datadog.configure do |config|
    config.tracing.instrument(:rails, service_name: Sidekiq.server? ? 'sidekiq-rails' : 'api-rails')
  end
  
  Rails.application.config.after_initialize do
    Datadog.configure do |config|
      config.tracing.instrument(:sidekiq, service_name: "#{config.service}-sidekiq")
      ....
    end
  end

[delner]

Ah yes, you're right @olkeene... I read the original example too hastily.

Datadog.configure & c.tracing.instrument :rails must run during the initialization phase for Rails, so it can modify the Rack stack. (This happens by default if you just add it to an initializer file.) By putting things in a Rails.application.config.to_prepare do block, you may be deferring this setup until after the application has already been initialized.

@drqCode Is there any particular reason why you placed this in the Rails.application.config.to_prepare do block in the first place?

[mbarany]

I had the same issue. I was following this zeitwerk migration guide which tells you to wrap all initializers with a Rails.application.config.to_prepare do block.

When I removed it the check bin/rails zeitwerk:check still passed with the message All is good! and my datadog APM was working again.

[marcotc]

This errors puts a limitation on ddtrace on how late the Datadog.configure block can be invoked safely:

Failed to apply Datadog::Contrib::Rails::Patcher patch. Cause: can't modify frozen Array
Location: /usr/local/bundle/gems/actionpack-7.0.3/lib/action_dispatch/middleware/stack.rb:102:in `insert

When ddtrace tries to inject its Rails middleware, it fails after the application has started, and thus the middleware stack is frozen.

We should document this in our Rails onboarding section, as well as rescue this specific error and output an actionable set of instructions on how to move the Datadog.configure block to a place where it is safe to invoke.