Invalid UTF-8 in breadcrumb messages cause events to not be sent
Issue Description
Sometimes breadcrumbs with a message that is not valid UTF-8 is generated (for example when using libvips) which results in the transaction not being sent.
Event sending failed: source sequence is illegal/malformed utf-8
~/ruby/3.2.4/lib/ruby/3.2.0/json/common.rb:312:in `generate'
~/ruby/3.2.4/lib/ruby/3.2.0/json/common.rb:312:in `generate'
~/ruby/gems/sentry-ruby-5.17.3/lib/sentry/envelope.rb:39:in `to_s'
~/ruby/gems/sentry-ruby-5.17.3/lib/sentry/envelope.rb:43:in `serialize'
~/ruby/gems/sentry-ruby-5.17.3/lib/sentry/transport.rb:74:in `block in serialize_envelope'
~/ruby/gems/sentry-ruby-5.17.3/lib/sentry/transport.rb:73:in `each'
~/ruby/gems/sentry-ruby-5.17.3/lib/sentry/transport.rb:73:in `serialize_envelope'
~/ruby/gems/sentry-ruby-5.17.3/lib/sentry/transport.rb:61:in `send_envelope'
~/ruby/gems/sentry-ruby-5.17.3/lib/sentry/transport.rb:51:in `send_event'
~/ruby/gems/sentry-ruby-5.17.3/lib/sentry/client.rb:192:in `send_event'
~/ruby/gems/sentry-ruby-5.17.3/lib/sentry/client.rb:254:in `block in dispatch_background_event'
~/ruby/gems/sentry-ruby-5.17.3/lib/sentry/background_worker.rb:76:in `_perform'
~/ruby/gems/sentry-ruby-5.17.3/lib/sentry/background_worker.rb:56:in `block in perform'
Related issues include #1992 and #1911, the difference here is that the malformed UTF-8 is present in the breadcrumb messages and not in redis keys or spans. If possible, it would be great with a check that detects and handles malformed UTF-8 across the entire transaction event.
Reproduction Steps
e = Sentry::Event.new(configuration: Sentry.configuration)
b = Sentry::BreadcrumbBuffer.new
b.record(Sentry::Breadcrumb.new(message: "\x76\xab\x99\xb5"))
e.breadcrumbs = b
Sentry.send_event(e, {})
Expected Behavior
The transaction event is sent
Actual Behavior
Event sending failed: source sequence is illegal/malformed utf-8
Ruby Version
3.2.0
SDK Version
5.17.3
Integration and Its Version
Rails 7.0.8
Sentry Config
Sentry.init do |config|
# ...
config.traces_sample_rate = 1.0
config.breadcrumbs_logger = [:sentry_logger, :monotonic_active_support_logger]
end
I agree to filter out binary is a better fix.
I want to share some more information. I made a monkeypatch for this issue.
def to_hash
breadcrumbs&.each do |b|
next if b.nil?
# Vips binary input is not serializable
# ignore binary message
b.message = '' if b&.category&.start_with?('Vips::')
end
super
end
end
This works for SDK 5.19.0 in production.