sentry-ruby icon indicating copy to clipboard operation
sentry-ruby copied to clipboard

Vernier + profiling enabled causing silent crashes

Open adam12 opened this issue 10 months ago • 2 comments

Issue Description

With Vernier installed and configured as the profiler, there is a random segmentation fault encoutered inside Sentry, causing a 500 error user-facing, but no exception ever reported to Sentry. This can cause silent failures where there are no visible signs that the website is flakey other than failed requests.

I had this manifest on gemdocs.org.

From Gemfile.lock

    concurrent-ruby (1.3.5)
    sentry-rails (5.22.4)
      railties (>= 5.0)
      sentry-ruby (~> 5.22.4)
    sentry-ruby (5.22.4)
      bigdecimal
      concurrent-ruby (~> 1.0, >= 1.0.2)
    vernier (1.5.0)
Stacktrace

[9403b-a7c0-4727-b91c-e3518d748965] Started GET "/gems/datadog_api_client/2.23.0/DatadogAPIClient/V2/RUMApplicationCreateRequest.html" for 37.19.207.34 at 2025-02-23 04:27:54 +0000
/app/vendor/bundle/ruby/3.4.0/gems/vernier-1.5.0/lib/vernier/collector.rb:77: [BUG] Segmentation fault at 0x0000000000000000
ruby 3.4.1 (2024-12-25 revision 48d4efcb85) +YJIT +PRISM [x86_64-linux]

-- Control frame information ----------------------------------------------- c:0036 p:---- s:0243 e:000242 CFUNC :finish c:0035 p:0003 s:0239 e:000238 METHOD /app/vendor/bundle/ruby/3.4.0/gems/vernier-1.5.0/lib/vernier/collector.rb:77 c:0034 p:0016 s:0231 e:000230 METHOD /app/vendor/bundle/ruby/3.4.0/gems/vernier-1.5.0/lib/vernier.rb:56 c:0033 p:0018 s:0226 e:000225 METHOD /app/vendor/bundle/ruby/3.4.0/gems/sentry-ruby-5.22.4/lib/sentry/vernier/profiler.rb:76 c:0032 p:0043 s:0221 e:000220 METHOD /app/vendor/bundle/ruby/3.4.0/gems/sentry-ruby-5.22.4/lib/sentry/transaction.rb:260 c:0031 p:0017 s:0211 e:000210 METHOD /app/vendor/bundle/ruby/3.4.0/gems/sentry-ruby-5.22.4/lib/sentry/rack/capture_exceptions.rb:83 c:0030 p:0102 s:0205 e:000204 BLOCK /app/vendor/bundle/ruby/3.4.0/gems/sentry-ruby-5.22.4/lib/sentry/rack/capture_exceptions.rb:43 c:0029 p:0020 s:0198 e:000197 METHOD /app/vendor/bundle/ruby/3.4.0/gems/sentry-ruby-5.22.4/lib/sentry/hub.rb:269 c:0028 p:0017 s:0193 e:000192 METHOD /app/vendor/bundle/ruby/3.4.0/gems/sentry-ruby-5.22.4/lib/sentry-ruby.rb:420 c:0027 p:0006 s:0188 e:000187 BLOCK /app/vendor/bundle/ruby/3.4.0/gems/sentry-ruby-5.22.4/lib/sentry/rack/capture_exceptions.rb:21 c:0026 p:0009 s:0184 e:000183 METHOD /app/vendor/bundle/ruby/3.4.0/gems/sentry-ruby-5.22.4/lib/sentry/hub.rb:59 c:0025 p:0017 s:0179 e:000178 METHOD /app/vendor/bundle/ruby/3.4.0/gems/sentry-ruby-5.22.4/lib/sentry-ruby.rb:400 c:0024 p:0024 s:0174 e:000173 METHOD /app/vendor/bundle/ruby/3.4.0/gems/sentry-ruby-5.22.4/lib/sentry/rack/capture_exceptions.rb:20 c:0023 p:0007 s:0169 e:000168 METHOD /app/vendor/bundle/ruby/3.4.0/gems/actionpack-8.0.1/lib/action_dispatch/middleware/show_exceptions.rb:32 c:0022 p:0047 s:0160 E:001a38 METHOD /app/vendor/bundle/ruby/3.4.0/gems/railties-8.0.1/lib/rails/rack/logger.rb:41 c:0021 p:0048 s:0147 e:000146 METHOD /app/vendor/bundle/ruby/3.4.0/gems/railties-8.0.1/lib/rails/rack/logger.rb:29 c:0020 p:0028 s:0141 e:000140 METHOD /app/vendor/bundle/ruby/3.4.0/gems/railties-8.0.1/lib/rails/rack/silence_request.rb:28 c:0019 p:0034 s:0136 e:000135 METHOD /app/vendor/bundle/ruby/3.4.0/gems/actionpack-8.0.1/lib/action_dispatch/middleware/remote_ip.rb:96 c:0018 p:0030 s:0130 e:000129 METHOD /app/vendor/bundle/ruby/3.4.0/gems/actionpack-8.0.1/lib/action_dispatch/middleware/request_id.rb:34 c:0017 p:0057 s:0124 e:000123 METHOD /app/vendor/bundle/ruby/3.4.0/gems/rack-3.1.9/lib/rack/method_override.rb:28 c:0016 p:0013 s:0118 e:000117 METHOD /app/vendor/bundle/ruby/3.4.0/gems/rack-3.1.9/lib/rack/runtime.rb:24 c:0015 p:0019 s:0108 e:000107 METHOD /app/vendor/bundle/ruby/3.4.0/gems/activesupport-8.0.1/lib/active_support/cache/strategy/local_cache_middleware.rb:29 c:0014 p:0016 s:0101 e:000100 METHOD /app/vendor/bundle/ruby/3.4.0/gems/actionpack-8.0.1/lib/action_dispatch/middleware/executor.rb:16 c:0013 p:0018 s:0092 e:000091 METHOD /app/vendor/bundle/ruby/3.4.0/gems/actionpack-8.0.1/lib/action_dispatch/middleware/static.rb:27 c:0012 p:0007 s:0087 e:000086 METHOD /app/vendor/bundle/ruby/3.4.0/gems/rack-3.1.9/lib/rack/sendfile.rb:114 c:0011 p:0021 s:0074 e:000073 METHOD /app/vendor/bundle/ruby/3.4.0/gems/actionpack-8.0.1/lib/action_dispatch/middleware/ssl.rb:92 c:0010 p:0043 s:0068 e:000067 METHOD /app/vendor/bundle/ruby/3.4.0/gems/actionpack-8.0.1/lib/action_dispatch/middleware/assume_ssl.rb:24 c:0009 p:0016 s:0063 e:000062 METHOD /app/vendor/bundle/ruby/3.4.0/gems/railties-8.0.1/lib/rails/engine.rb:535 c:0008 p:0017 s:0057 e:000056 METHOD /app/vendor/bundle/ruby/3.4.0/gems/puma-6.6.0/lib/puma/configuration.rb:279 c:0007 p:0007 s:0052 e:000051 BLOCK /app/vendor/bundle/ruby/3.4.0/gems/puma-6.6.0/lib/puma/request.rb:99 c:0006 p:0015 s:0049 e:000048 METHOD /app/vendor/bundle/ruby/3.4.0/gems/puma-6.6.0/lib/puma/thread_pool.rb:390 c:0005 p:0353 s:0044 e:000043 METHOD /app/vendor/bundle/ruby/3.4.0/gems/puma-6.6.0/lib/puma/request.rb:98 c:0004 p:0093 s:0029 e:000028 METHOD /app/vendor/bundle/ruby/3.4.0/gems/puma-6.6.0/lib/puma/server.rb:472 c:0003 p:0005 s:0018 e:000017 BLOCK /app/vendor/bundle/ruby/3.4.0/gems/puma-6.6.0/lib/puma/server.rb:254 c:0002 p:0071 s:0014 e:000013 BLOCK /app/vendor/bundle/ruby/3.4.0/gems/puma-6.6.0/lib/puma/thread_pool.rb:167 [FINISH] c:0001 p:---- s:0003 e:000002 DUMMY [FINISH]

-- Ruby level backtrace information ---------------------------------------- /app/vendor/bundle/ruby/3.4.0/gems/puma-6.6.0/lib/puma/thread_pool.rb:167:in 'block in spawn_thread' /app/vendor/bundle/ruby/3.4.0/gems/puma-6.6.0/lib/puma/server.rb:254:in 'block in run' /app/vendor/bundle/ruby/3.4.0/gems/puma-6.6.0/lib/puma/server.rb:472:in 'process_client' /app/vendor/bundle/ruby/3.4.0/gems/puma-6.6.0/lib/puma/request.rb:98:in 'handle_request' /app/vendor/bundle/ruby/3.4.0/gems/puma-6.6.0/lib/puma/thread_pool.rb:390:in 'with_force_shutdown'

/pre>

Reproduction Steps

Unknown.

Expected Behavior

Not to crash.

Actual Behavior

Crashes silently.

Ruby Version

3.4.1

SDK Version

5.22.4

Integration and Its Version

Rails (8.0.1)

Sentry Config

Sentry.init do |config|
  config.dsn = ENV["SENTRY_DSN"]

  # get breadcrumbs from logs
  config.breadcrumbs_logger = [:active_support_logger, :http_logger]

  # enable profiling
  # this is relative to traces_sample_rate
  config.profiles_sample_rate = 1.0

  # Use Vernier for profiling
  config.profiler_class = Sentry::Vernier::Profiler

  config.traces_sampler = lambda do |sampling_context|
    # if this is the continuation of a trace, just use that decision (rate controlled by the caller)
    unless sampling_context[:parent_sampled].nil?
      next sampling_context[:parent_sampled]
    end

    0.5
  end
end

adam12 avatar Mar 06 '25 22:03 adam12

Thanks for reporting this Adam 🙇🏻 I'll be looking into this today, but since this is coming from Vernier I'm afraid I won't be able to fix it quickly. One thing I need to check is why segfaults are not being reported for you by Sentry.

solnic avatar Mar 07 '25 11:03 solnic

we are simply a library that runs within ruby, segfaults crash ruby entirely so there is no way we could catch those. That would require an agent living parallel to the ruby process.

sl0thentr0py avatar Mar 13 '25 15:03 sl0thentr0py

don't think we can do anything here, closing

sl0thentr0py avatar Sep 16 '25 10:09 sl0thentr0py