sentry-ruby icon indicating copy to clipboard operation
sentry-ruby copied to clipboard

Stack level to deep error after upgrading to 4.4.0

Open st0012 opened this issue 3 years ago • 25 comments

In the version 4.4.0, we added support for capturing net/http requests as span/breadcrumbs. This feature requires patching the class, which can be done in 2 different but conflicting ways: method aliasing and module prepending. Here's a great article from bearer explaining how these 2 approaches conflict with each other.

Fortunately, the community has been adopting the prepend approach for a while now + sentry-ruby requires Ruby 2.4+. So the chance of having the issue should be low.

How we're going to help you upgrade

With that being said, if you're reading this issue, chances are you're having the conflict inside your app. So here's our plan to help you upgrade:

  1. We'll see if it's possible to align your other dependencies toward the prepend approach. Most of the gems that patch this library should have provided an option to switch the mode to prepend. For example, rack-mini-profiler and its prepend_net_http_patch option.
  2. If the above option is not viable to you, we'll decide whether which approach to take based on your feedback:
    • Provide a way to opt-out this patch.
    • Provide an option to switch modes like some other gems do.

Update

The version 4.4.1 should solve the issue.

st0012 avatar May 04 '21 01:05 st0012

I just checked scout_apm's implementation and it looks like users can only solve the problem by disabling their net/http instrumentation. I also checked their instrumentation implementations, which mostly (if not all) use the alias approach.

Consider scout_apm is one of the most popular apm services in the community, I will add an option for users to switch between prepend and alias mode. Something like: prepend_net_instrumentation: boolean.

st0012 avatar May 05 '21 06:05 st0012

I think changing the timing of prepend could also solve the problem without introducing another patching method. I've implemented it in https://github.com/getsentry/sentry-ruby/pull/1432 and will cut a release soon.

st0012 avatar May 05 '21 15:05 st0012

sentry-ruby 4.4.1 is out now, which should solve the issue. Sorry for the inconvenience.

st0012 avatar May 05 '21 16:05 st0012

I was hit by similar problem with redis instrumentation between opentelemetry and scout. they have migrated to using prepend in redis, but this issue tells me they should do it everywhere...

mjobin-mdsol avatar May 10 '21 04:05 mjobin-mdsol

FYI: This is very much not fixed in 4.4.1. We just tried a production deploy including v4.4.1 last night and this upgrade inadvertently took our entire site down.

Not sure how much it will help, but see attached log.

mvastola avatar May 11 '21 08:05 mvastola

@mvastola I'm really sorry that it still causes you the problem 😔 But I can't reproduce the issue with 4.4.1 with my own app or the SDK's example app. Can you create an example repo that can reproduce the issue? That'll help me identify and fix the issue more quickly.

st0012 avatar May 12 '21 04:05 st0012

Same issue:

2021-05-11T09:21:04.562282+00:00 app[web.1]: [Scout] [05/11/21 09:21:03 +0000 web.1 (29)] INFO : Posting payload to "https://checkin.scoutapp.com"
2021-05-11T09:21:04.562283+00:00 app[web.1]: [Scout] [05/11/21 09:21:04 +0000 web.1 (29)] INFO : Exception sending request to server:
2021-05-11T09:21:04.562284+00:00 app[web.1]: stack level too deep
2021-05-11T09:21:04.562284+00:00 app[web.1]: /app/vendor/bundle/ruby/2.7.0/gems/scout_apm-4.0.4/lib/scout_apm/config.rb:226:in `each'
2021-05-11T09:21:04.562285+00:00 app[web.1]: /app/vendor/bundle/ruby/2.7.0/gems/scout_apm-4.0.4/lib/scout_apm/config.rb:226:in `detect'
2021-05-11T09:21:04.562285+00:00 app[web.1]: /app/vendor/bundle/ruby/2.7.0/gems/scout_apm-4.0.4/lib/scout_apm/config.rb:226:in `overlay_for_key'
2021-05-11T09:21:04.562286+00:00 app[web.1]: /app/vendor/bundle/ruby/2.7.0/gems/scout_apm-4.0.4/lib/scout_apm/config.rb:234:in `value'
2021-05-11T09:21:04.562288+00:00 app[web.1]: /app/vendor/bundle/ruby/2.7.0/gems/scout_apm-4.0.4/lib/scout_apm/instruments/net_http.rb:43:in `request_scout_description'
2021-05-11T09:21:04.562289+00:00 app[web.1]: /app/vendor/bundle/ruby/2.7.0/gems/scout_apm-4.0.4/lib/scout_apm/instruments/net_http.rb:29:in `request_with_scout_instruments'
2021-05-11T09:21:04.562289+00:00 app[web.1]: /app/vendor/bundle/ruby/2.7.0/gems/sentry-ruby-core-4.4.1/lib/sentry/net/http.rb:9:in `request'
2021-05-11T09:21:04.562290+00:00 app[web.1]: /app/vendor/bundle/ruby/2.7.0/gems/scout_apm-4.0.4/lib/scout_apm/instruments/net_http.rb:30:in `block in request_with_scout_instruments'
...
...
2021-05-11T09:21:04.589514+00:00 app[web.1]: /app/vendor/bundle/ruby/2.7.0/gems/scout_apm-4.0.4/lib/scout_apm/instruments/net_http.rb:30:in `block in request_with_scout_instruments'
2021-05-11T09:21:04.589518+00:00 app[web.1]: /app/vendor/bundle/ruby/2.7.0/gems/scout_apm-4.0.4/lib/scout_apm/tracer.rb:34:in `instrument'
2021-05-11T09:21:04.589518+00:00 app[web.1]: /app/vendor/bundle/ruby/2.7.0/gems/scout_apm-4.0.4/lib/scout_apm/tracer.rb:44:in `instrument'
2021-05-11T09:21:04.589518+00:00 app[web.1]: /app/vendor/bundle/ruby/2.7.0/gems/scout_apm-4.0.4/lib/scout_apm/instruments/net_http.rb:29:in `request_with_scout_instruments'
2021-05-11T09:21:04.589519+00:00 app[web.1]: /app/vendor/bundle/ruby/2.7.0/gems/sentry-ruby-core-4.4.1/lib/sentry/net/http.rb:9:in `request'
2021-05-11T09:21:04.589519+00:00 app[web.1]: /app/vendor/bundle/ruby/2.7.0/gems/scout_apm-4.0.4/lib/scout_apm/instruments/net_http.rb:30:in `block in request_with_scout_instruments'
2021-05-11T09:21:04.589519+00:00 app[web.1]: /app/vendor/bundle/ruby/2.7.0/gems/scout_apm-4.0.4/lib/scout_apm/tracer.rb:34:in `instrument'
2021-05-11T09:21:04.589528+00:00 app[web.1]: /app/vendor/bundle/ruby/2.7.0/gems/scout_apm-4.0.4/lib/scout_apm/tracer.rb:44:in `instrument'
2021-05-11T09:21:04.589528+00:00 app[web.1]: /app/vendor/bundle/ruby/2.7.0/gems/scout_apm-4.0.4/lib/scout_apm/instruments/net_http.rb:29:in `request_with_scout_instruments'
2021-05-11T09:21:04.589529+00:00 app[web.1]: /app/vendor/bundle/ruby/2.7.0/gems/sentry-ruby-core-4.4.1/lib/sentry/net/http.rb:9:in `request'
2021-05-11T09:21:04.589529+00:00 app[web.1]: /app/vendor/bundle/ruby/2.7.0/gems/scout_apm-4.0.4/lib/scout_apm/instruments/net_http.rb:30:in `block in request_with_scout_instruments'
2021-05-11T09:21:04.589529+00:00 app[web.1]: /app/vendor/bundle/ruby/2.7.0/gems/scout_apm-4.0.4/lib/scout_apm/tracer.rb:34:in `instrument'
2021-05-11T09:21:04.589529+00:00 app[web.1]: /app/vendor/bundle/ruby/2.7.0/gems/scout_apm-4.0.4/lib/scout_apm/tracer.rb:44:in `instrument'
2021-05-11T09:21:04.589530+00:00 app[web.1]: /app/vendor/bundle/ruby/2.7.0/gems/scout_apm-4.0.4/lib/scout_apm/instruments/net_http.rb:29:in `request_with_scout_instruments'
2021-05-11T09:21:04.589530+00:00 app[web.1]: /app/vendor/bundle/ruby/2.7.0/gems/sentry-ruby-core-4.4.1/lib/sentry/net/http.rb:9:in `request'
2021-05-11T09:21:04.589530+00:00 app[web.1]: /app/vendor/bundle/ruby/2.7.0/gems/scout_apm-4.0.4/lib/scout_apm/instruments/net_http.rb:30:in `block in request_with_scout_instruments'
2021-05-11T09:21:04.589530+00:00 app[web.1]: /app/vendor/bundle/ruby/2.7.0/gems/scout_apm-4.0.4/lib/scout_apm/tracer.rb:34:in `instrument'
2021-05-11T09:21:04.589531+00:00 app[web.1]: /app/vendor/bundle/ruby/2.7.0/gems/scout_apm-4.0.4/lib/scout_apm/tracer.rb:44:in `instrument'
2021-05-11T09:21:04.589531+00:00 app[web.1]: /app/vendor/bundle/ruby/2.7.0/gems/scout_apm-4.0.4/lib/scout_apm/instruments/net_http.rb:29:in `request_with_scout_instruments'
2021-05-11T09:21:04.589531+00:00 app[web.1]: /app/vendor/bundle/ruby/2.7.0/gems/sentry-ruby-core-4.4.1/lib/sentry/net/http.rb:9:in `request'
2021-05-11T09:21:04.589531+00:00 app[web.1]: /app/vendor/bundle/ruby/2.7.0/gems/scout_apm-4.0.4/lib/scout_apm/reporter.rb:85:in `block in post'
2021-05-11T09:21:04.589532+00:00 app[web.1]: /app/vendor/ruby-2.7.3/lib/ruby/2.7.0/net/http.rb:933:in `start'
2021-05-11T09:21:04.589533+00:00 app[web.1]: /app/vendor/bundle/ruby/2.7.0/gems/scout_apm-4.0.4/lib/scout_apm/reporter.rb:92:in `request'
2021-05-11T09:21:04.589533+00:00 app[web.1]: /app/vendor/bundle/ruby/2.7.0/gems/scout_apm-4.0.4/lib/scout_apm/reporter.rb:80:in `post'
2021-05-11T09:21:04.589533+00:00 app[web.1]: /app/vendor/bundle/ruby/2.7.0/gems/scout_apm-4.0.4/lib/scout_apm/reporter.rb:158:in `block in post_payload'
2021-05-11T09:21:04.589533+00:00 app[web.1]: /app/vendor/bundle/ruby/2.7.0/gems/scout_apm-4.0.4/lib/scout_apm/reporter.rb:156:in `each'
2021-05-11T09:21:04.589534+00:00 app[web.1]: /app/vendor/bundle/ruby/2.7.0/gems/scout_apm-4.0.4/lib/scout_apm/reporter.rb:156:in `post_payload'
2021-05-11T09:21:04.589534+00:00 app[web.1]: /app/vendor/bundle/ruby/2.7.0/gems/scout_apm-4.0.4/lib/scout_apm/reporter.rb:40:in `report'
2021-05-11T09:21:04.589534+00:00 app[web.1]: /app/vendor/bundle/ruby/2.7.0/gems/scout_apm-4.0.4/lib/scout_apm/reporting.rb:93:in `deliver_period'
2021-05-11T09:21:04.589535+00:00 app[web.1]: /app/vendor/bundle/ruby/2.7.0/gems/scout_apm-4.0.4/lib/scout_apm/reporting.rb:52:in `block in report_to_server'
2021-05-11T09:21:04.589535+00:00 app[web.1]: /app/vendor/bundle/ruby/2.7.0/gems/scout_apm-4.0.4/lib/scout_apm/layaway.rb:103:in `with_claim'
2021-05-11T09:21:04.589535+00:00 app[web.1]: /app/vendor/bundle/ruby/2.7.0/gems/scout_apm-4.0.4/lib/scout_apm/reporting.rb:45:in `report_to_server'
2021-05-11T09:21:04.589536+00:00 app[web.1]: /app/vendor/bundle/ruby/2.7.0/gems/scout_apm-4.0.4/lib/scout_apm/reporting.rb:36:in `process_metrics'
2021-05-11T09:21:04.589536+00:00 app[web.1]: /app/vendor/bundle/ruby/2.7.0/gems/scout_apm-4.0.4/lib/scout_apm/periodic_work.rb:13:in `run'
2021-05-11T09:21:04.589536+00:00 app[web.1]: /app/vendor/bundle/ruby/2.7.0/gems/scout_apm-4.0.4/lib/scout_apm/agent.rb:178:in `block (2 levels) in start_background_worker'
2021-05-11T09:21:04.589537+00:00 app[web.1]: /app/vendor/bundle/ruby/2.7.0/gems/scout_apm-4.0.4/lib/scout_apm/background_worker.rb:61:in `block in start'
2021-05-11T09:21:04.589537+00:00 app[web.1]: /app/vendor/bundle/ruby/2.7.0/gems/scout_apm-4.0.4/lib/scout_apm/background_worker.rb:44:in `loop'
2021-05-11T09:21:04.589537+00:00 app[web.1]: /app/vendor/bundle/ruby/2.7.0/gems/scout_apm-4.0.4/lib/scout_apm/background_worker.rb:44:in `start'
2021-05-11T09:21:04.589537+00:00 app[web.1]: /app/vendor/bundle/ruby/2.7.0/gems/scout_apm-4.0.4/lib/scout_apm/agent.rb:177:in `block in start_background_worker'

RobWu avatar May 12 '21 07:05 RobWu

@RobWu @mvastola are you apps using Rails? It'd be helpful to know this information because I think I've found the cause.

If it's not a Rails app, can you make this change: make sure Sentry.init is only called after require "scout_apm" in your app?

If it is a Rails app, can you provide me the Rails version + your scout/sentry configs? thanks

st0012 avatar May 12 '21 08:05 st0012

@st0012, see below. Part of the insidiousness of this though is that this issue only emerged in prod. (I think it was compounded by the fact that errors happen relatively more frequently in prod, and a stack overflow maxes out memory, so everything else either takes forever or fails.)

We don't manually require 'scout_apm' at all, but we could theoretically do so above our Sentry.init. Only problem is we don't plan on deploying this version again (for obvious reasons), so I couldn't tell you if this worked.

* rails (6.1.3.2)
    Summary: Full-stack web application framework.
    Homepage: https://rubyonrails.org
    Path: /usr/local/rvm/gems/ruby-2.7.2/gems/rails-6.1.3.2
* scout_apm (2.6.3)
    Summary: Ruby application performance monitoring
    Homepage: https://github.com/scoutapp/scout_apm_ruby
    Path: /usr/local/rvm/gems/ruby-2.7.2/gems/scout_apm-2.6.3
* sentry-rails (4.4.0)
    Summary: A gem that provides Rails integration for the Sentry error logger
    Homepage: https://github.com/getsentry/sentry-ruby
    Path: /usr/local/rvm/gems/ruby-2.7.2/gems/sentry-rails-4.4.0
* sentry-ruby (4.4.1)
    Summary: A gem that provides a client interface for the Sentry error logger
    Homepage: https://github.com/getsentry/sentry-ruby
    Path: /usr/local/rvm/gems/ruby-2.7.2/gems/sentry-ruby-4.4.1
* sentry-ruby-core (4.4.1)
   Summary: A gem that provides a client interface for the Sentry error logger
   Homepage: https://github.com/getsentry/sentry-ruby
   Path: /usr/local/rvm/gems/ruby-2.7.2/gems/sentry-ruby-core-4.4.1

config/initializers/sentry.rb:

if defined?(Sentry)
    Sentry.init do |config|
        config.dsn = Rails.application.credentials.sentry[:ruby_dsn]
        config.breadcrumbs_logger = [:active_support_logger]
        config.environment = Rails.env
        # config.backtrace_cleanup_callback = lambda { |backtrace| Rails.backtrace_cleaner.clean(backtrace) }
        config.release = File.read('RELEASE_VERSION').chomp
    end
end

config/scout_apm.yml:

common: &defaults
  key: [REMOVED]
  monitor: true
  ignore:
    - '/webhooks/zendesk'
    - '/404'
production:
  <<: *defaults
  auto_instruments: true

mvastola avatar May 12 '21 15:05 mvastola

@mvastola thanks for sharing those info. unfortunately, I still can't reproduce the problem with the same configs. however, I think you can reproduce the issue locally by changing this

common: &defaults
  key: [REMOVED]
  monitor: true
  ignore:
    - '/webhooks/zendesk'
    - '/404'
- production:
+ development:
  <<: *defaults
  auto_instruments: true

btw, I don't recommend initializing the SDK conditionally. you can use the config.enabled_environments option to control the SDK's activation, like

Sentry.init do |config|
  config.dsn = Rails.application.credentials.sentry[:ruby_dsn]
  config.breadcrumbs_logger = [:active_support_logger]
  config.environment = Rails.env
  config.enabled_environments = %w[production]
  # config.backtrace_cleanup_callback = lambda { |backtrace| Rails.backtrace_cleaner.clean(backtrace) }
  config.release = File.read('RELEASE_VERSION').chomp
end

st0012 avatar May 13 '21 08:05 st0012

There is an open PR on the scout_apm gem which has flip the Net/HTTP instrumentation to use Module#prepend, I would try that next.

mjobin-mdsol avatar May 14 '21 01:05 mjobin-mdsol

# config/scout_apm.yml
common: &defaults
  disabled_instruments:
    - "NetHttp"

solves the issue for me

My config:

Sentry.init do |config|
  config.dsn = Rails.application.credentials[:sentry][:dsn]
  config.enabled_environments = %w[production staging]
  config.breadcrumbs_logger = [:sentry_logger] 
  config.excluded_exceptions = []
end

RobWu avatar Jun 18 '21 08:06 RobWu

So this is still very much broken despite upgrading to the latest versions of scout_apm and sentry-*.

Our latest backtrace is here. I turned on some debugging information, which might make things easier to trace.

I was finally able to reproduce the problem on a non-production instance, and -- trying to debug this a bit -- it looks like the execution flow is this:

The request method in Net:HTTP is defined:

irb(main):004:0> Net::HTTP.instance_method(:request)
=> #<UnboundMethod: Net::HTTP(Sentry::Net::HTTP)#request(req, body=..., &block) /usr/local/bundle/gems/sentry-ruby-core-4.5.1/lib/sentry/net/http.rb:35>

.. and that #request method calls its super, which is:

irb(main):006:0> Net::HTTP.instance_method(:request).super_method
=> #<UnboundMethod: Net::HTTP#request(request_with_scout_instruments)(*args, &block) /usr/local/bundle/gems/scout_apm-4.1.1/lib/scout_apm/instruments/net_http.rb:28>

.. that method, Net::HTTP.request_with_scout_instruments, then calls request_without_scout_instruments, which is:

irb(main):003:0> Net::HTTP.instance_method(:request_without_scout_instruments)
=> #<UnboundMethod: Net::HTTP(Sentry::Net::HTTP)#request_without_scout_instruments(request)(req, body=..., &block) /usr/local/bundle/gems/sentry-ruby-core-4.5.1/lib/sentry/net/http.rb:35>

.. thereby creating the infinite loop.

I am not able to reproduce on a brand new rails app though, so I imagine the issue could just be with order in which these hooks are applied.

mvastola avatar Jun 20 '21 01:06 mvastola

Update: scout_apm mentioned that they plan to switch to the prepend approach too in the upcoming 5.0 version:

I am prepping a 4.1.0 version of the agent with several bug fixes, and then after that looking at 5.0. The big breaking change is switching to all prepend by default, and adding a configuration option to use alias if that causes compatibility issues with other gems.

While every ruby we support also has prepend, many other gems use alias_method, and that causes issues when mixed. Keep an eye out for this change, should be up pretty quick.

https://github.com/scoutapp/scout_apm_ruby/pull/402#issuecomment-846480983

st0012 avatar Jul 28 '21 09:07 st0012

@st0012 good catch. Thanks. Just not sure when "pretty quick" is since that comment you linked is from May. :slightly_frowning_face:

mvastola avatar Jul 28 '21 17:07 mvastola

@mvastola 4.1.1 solved the issues I had, so on my end, I'm Ok with the current state of things

mjobin-mdsol avatar Aug 23 '21 05:08 mjobin-mdsol

@st0012 I ran into the same issue due to the gem skylight. This comment helped.

SystemStackError(stack level too deep) reproduction script:

require 'bundler/inline'

gemfile do
  source 'https://rubygems.org' do
    gem 'rails', require: true
    gem 'sentry-ruby'
    gem 'skylight', '< 5.0.0'     # SystemStackError
    # gem 'skylight', '>= 5.0.0'    # No SystemStackError
  end
end

class TestApp < Rails::Application
  Sentry.init do; end
end

TestApp.initialize!

uri = URI('http://github.com')
http = Net::HTTP.new(uri.host, uri.port)
http.set_debug_output($stdout)

http.request(Net::HTTP::Get.new(uri.request_uri))

Update: Looks like catching the aliasing is not straight-forward.

adi-pen avatar Aug 24 '21 01:08 adi-pen

So this is still very much broken despite upgrading to the latest versions of scout_apm and sentry-*. Stack trace:

","name":"Rails","message":"sentry -- Sending envelope [event] d7b9aec42f40494e9b13a0fa62616fbb to Sentry"}
[1mTraceback[m (most recent call last):
	9634: from /app/bin/rails:5:in `<main>'
	9633: from /app/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.9.1/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:31:in `require'
	9632: from /app/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.9.1/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:22:in `require_with_bootsnap_lfi'
	9631: from /app/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.9.1/lib/bootsnap/load_path_cache/loaded_features_index.rb:92:in `register'
	9630: from /app/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.9.1/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `block in require_with_bootsnap_lfi'
	9629: from /app/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.9.1/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `require'
	9628: from /app/vendor/bundle/ruby/2.7.0/gems/railties-6.1.4.1/lib/rails/commands.rb:18:in `<main>'
	9627: from /app/vendor/bundle/ruby/2.7.0/gems/railties-6.1.4.1/lib/rails/command.rb:50:in `invoke'
	 ... 9622 levels...
	   4: from /app/vendor/bundle/ruby/2.7.0/gems/scout_apm-4.1.2/lib/scout_apm/instruments/net_http.rb:43:in `request_scout_description'
	   3: from /app/vendor/bundle/ruby/2.7.0/gems/scout_apm-4.1.2/lib/scout_apm/config.rb:237:in `value'
	   2: from /app/vendor/bundle/ruby/2.7.0/gems/scout_apm-4.1.2/lib/scout_apm/config.rb:229:in `overlay_for_key'
	   1: from /app/vendor/bundle/ruby/2.7.0/gems/scout_apm-4.1.2/lib/scout_apm/config.rb:229:in `detect'
/app/vendor/bundle/ruby/2.7.0/gems/scout_apm-4.1.2/lib/scout_apm/config.rb:229:in `each': [1mstack level too deep ([1;4mSystemStackError[m[1m)[m

Same problem over here. Still having this issue with scout_apm (4.1.2) and Sentry (4.7.3). Problem also occurs with Sentry 4.4.2, 4.5.2 and 4.6.5.

What is the recommended approach of action for now? Downgrade to Sentry 4.3 and wait for scout_apm 5 to land?

kreintjes avatar Sep 30 '21 08:09 kreintjes

@kreintjes I hope you (and others that still are bothered by the issue) to push Scout to migrate to the prepend approach. Since both skylight and appsignal have migrated to it in their recent releases too, I don't think there should be any technical blocker for this change (of course, it won't be an easy one).

And before they finally do it, you can disable Scout's http instrumentation to workaround the issue:

# config/scout_apm.yml
common: &defaults
  disabled_instruments:
    - "NetHttp"

st0012 avatar Oct 02 '21 04:10 st0012

This issue has gone three weeks without activity. In another week, I will close it.

But! If you comment or otherwise update it, I will reset the clock, and if you label it Status: Backlog or Status: In Progress, I will leave it alone ... forever!


"A weed is but an unloved flower." ― Ella Wheeler Wilcox 🥀

github-actions[bot] avatar Oct 28 '21 21:10 github-actions[bot]

Still tracking.

samsonjs avatar Oct 28 '21 21:10 samsonjs

What is the status on this?

We had our app crashing in prod because of this issue.

We had to disable scout entirely, because Sentry is more important on our stack.

jpaulomotta avatar May 13 '22 13:05 jpaulomotta

@jpaulomotta Sub-optimal, but a workaround to have both running since scout fix doesn't seem to be happening, at least this way

(this is for sentry 5.1.0 with scout 5.1.1)

  1. set a SCOUT_DISABLED_INSTRUMENTS ENV to ["Net::HTTP","NetHttp"]
  • i think only one of these is necessary, but i didn't go back and try to sort it out, took long enough as it was
  1. this might also help
  • it's been a while, but recall this not working for us so we did the ENV

mikescar avatar May 13 '22 14:05 mikescar

@jpaulomotta There's a glimmer of hope from Scout on this one but it's painfully slow going for some reason: https://github.com/scoutapp/scout_apm_ruby/pull/448

samsonjs avatar May 13 '22 15:05 samsonjs

Following @adi-pen's comment:

@st0012 I ran into the same issue due to the gem skylight. This comment helped.

I saw Skylight's v5 announcement seems relevant:

"Skylight 5 also includes a shift to Module#prepend for adding instrumentation to Rails"

Updating to Skylight v5 worked for us - required update to Ruby 2.7.3 however. (https://github.com/publiclab/mapknitter/pull/1712) (error log had been this)

jywarren avatar May 24 '22 20:05 jywarren

Given that https://github.com/scoutapp/scout_apm_ruby/pull/448 has been released in version 5.3.0. I'm closing this now.

If you have any similar issues, I recommend:

  1. Check if the conflicting package patches the component with alias_method
    • If it does, please file an issue there because they should migrate to prepend or at least provide an option for it.
  2. If it doesn't, check if the version you are using uses alias_method
    • If your version uses alias_method, please upgrade to a version that doesn't
    • If the version you're using doesn't use alias_method, but still causes this error with the SDK, please file a new issue

st0012 avatar Sep 03 '22 08:09 st0012