Support ruby 3.4

[gregawoods]

What does this pull request do?

This fixes the parsing of stack traces in the latest version of ruby.

Ruby 3.4 made a few changes to how traces are formatted:

1. The opening backtick is now a single quote
2. The module name is now included.

# ruby 3.3.6:
apm-agent-ruby/spec/support/exception_helpers.rb:22:in `/'

# ruby 3.4.1:
apm-agent-ruby/spec/support/exception_helpers.rb:22:in 'Integer#/'

These changes cause problems when the agent parses the string, which manifests as a validation error.

[ElasticAPM] [THREAD:14736]: Closing request with reason scheduled_flush
[ElasticAPM] APM Server responded with an error:
"{\"accepted\":2,\"errors\":[{\"message\":\"validation error: error: exception: stacktrace: requires at least one of the fields 'classname;filename'\",\"document\":\"{\\\"error\\\":{\\\"id\\\":\\\"[key here]\\\",\\\"transaction_id\\\":null,\\\"transaction\\\":null,\\\"trace_id\\\":null,\\\"parent_id\\\":null,\\\"culprit\\\":null,\\\"timestamp\\\":1735577232664940,\\\"exception\\\":{\\\"message\\\":\\\"divided by 0\\\",\\\"type\\\":\\\"ZeroDivisionError\\\",\\\"module\\\":\\\"\\\",\\\"code\\\":null,\\\"attributes\\\":null,\\\"stacktrace\\\":[{\\\"abs_path\\\":null,\\\"filename\\\":null,\\\"function\\\":null,\\\"lineno\\\":0,\\\"library_frame\\\":false},{\\\"abs_path\\\":null,\\\"filename\\\":null,\\\"function\\\":null,\\\"lineno\\\":0,\\\"library_frame\\\":false},{\\\"abs_path\\\":null,\\\"filename\\\":null,\\\"function\\\":null,\\\"lineno\\\":0,\\\"library_frame\\\":false},{\\\"abs_path\\\":null,\\\"filename\\\":null,\\\"function\\\":null,\\\"lineno\\\":0,\\\"library_frame\\\":false},{\\\"abs_path\\\":null,\\\"filename\\\":null,\\\"function\\\":null,\\\"lineno\\\":0,\\\"library_frame\\\":false},{\\\"abs_path\\\":null,\\\"filename\\\":null,\\\"function\\\":null,\\\"lineno\\\":0,\\\"library_frame\\\":false},{\\\"abs_path\\\":null,\\\"filename\\\":null,\\\"function\\\":null,\\\"lineno\\\":0,\\\"library_frame\\\":false},{\\\"abs_path\\\":null,\\\"filename\\\":null,\\\"function\\\":null,\\\"lineno\\\":0,\\\"library_frame\\\":false},{\\\"abs_path\\\":null,\\\"filename\\\":null,\\\"function\\\":null,\\\"lineno\\\":0,\\\"library_frame\\\":false},{\\\"abs_path\\\":null,\\\"filename\\\":null,\\\"function\\\":null,\\\"lineno\\\":0,\\\"library_frame\\\":false},{\\\"abs_path\\\":null,\\\"filename\\\":null,\\\"function\\\":null,\\\"lineno\\\":0,\\\"library_frame\\\":false},{\\\"abs_path\\\":null,\\\"filename\\\":null,\\\"function\\\":null,\\\"lineno\\\":0,\\\"library_frame\\\":false},{\\\"abs_path\\\":null,\\\"filename\\\":null,\\\"function\\\":null,\\\"lineno\\\":0,\\\"library_frame\\\":false},{\\\"abs_path\\\":null,\\\"filename\\\":null,\\\"function\\\":null,\\\"lineno\\\":0,\\\"library_frame\\\":false},{\\\"abs_path\\\":null,\\\"filename\\\":null,\\\"function\\\":null,\\\"lineno\\\":0,\\\"library_frame\\\":false},{\\\"abs_path\\\":null,\\\"filename\\\":null,\\\"function\\\":null,\\\"lineno\\\":0,\\\"library_frame\\\":false},{\\\"abs_path\\\":null,\\\"filename\\\":null,\\\"function\\\":null,\\\"lineno\\\":0,\\\"library_frame\\\":false},{\\\"abs_path\\\":null,\\\"filename\\\":null,\\\"function\\\":null,\\\"lineno\\\":0,\\\"library_frame\\\":false},{\\\"abs_path\\\":null,\\\"filename\\\":null,\\\"function\\\":null,\\\"lineno\\\":0,\\\"library_frame\\\":false},{\\\"abs_path\\\":null,\\\"filename\\\":null,\\\"function\\\":null,\\\"lineno\\\":0,\\\"library_frame\\\":false},{\\\"abs_path\\\":null,\\\"filename\\\":null,\\\"function\\\":null,\\\"lineno\\\":0,\\\"library_frame\\\":false},{\\\"abs_path\\\":null,\\\"filename\\\":null,\\\"function\\\":null,\\\"lineno\\\":0,\\\"library_frame\\\":false},{\\\"abs_path\\\":null,\\\"filename\\\":null,\\\"function\\\":null,\\\"lineno\\\":0,\\\"library_frame\\\":false},{\\\"abs_path\\\":null,\\\"filename\\\":null,\\\"function\\\":null,\\\"lineno\\\":0,\\\"library_frame\\\":false},{\\\"abs_path\\\":null,\\\"filename\\\":null,\\\"function\\\":null,\\\"lineno\\\":0,\\\"library_frame\\\":false},{\\\"abs_path\\\":null,\\\"filename\\\":null,\\\"function\\\":null,\\\"lineno\\\":0,\\\"library_frame\\\":false},{\\\"abs_path\\\":null,\\\"filename\\\":null,\\\"function\\\":null,\\\"lineno\\\":0,\\\"library_frame\\\":false},{\\\"abs_path\\\":null,\\\"filename\\\":null,\\\"function\\\":null,\\\"lineno\\\":0,\\\"library_frame\\\":false},{\\\"abs_path\\\":null,\\\"filename\\\":null,\\\"function\\\":null,\\\"lineno\\\":0,\\\"library_frame\\\":false},{\\\"abs_path\\\":null,\\\"filename\\\":null,\\\"function\\\":null,\\\"lineno\\\":0,\\\"library_frame\\\":false},{\\\"abs_path\\\":null,\\\"filename\\\":null,\\\"function\\\":null,\\\"lineno\\\":0,\\\"library_frame\\\":false},{\\\"abs_path\\\":null,\\\"filename\\\":null,\\\"function\\\":null,\\\"lineno\\\":0,\\\"library_frame\\\":false},{\\\"abs_path\\\":null,\\\"filename\\\":null,\\\"function\\\":null,\\\"lineno\\\":0,\\\"library_frame\\\":false}],\\\"handled\\\":true,\\\"cause\\\":null}}}\"}]}\n"

Why is it important?

This will block users of APM from upgrading to ruby 3.4.

Checklist

• [x] I have signed the Contributor License Agreement.
• [x] My code follows the style guidelines of this project (See .rubocop.yml)
• [x] I have rebased my changes on top of the latest main branch
• [x] I have added tests that prove my fix is effective or that my feature works
• [x] New and existing unit tests pass locally with my changes
• [x] I have made corresponding changes to the documentation
• [x] I have updated CHANGELOG.asciidoc
• [ ] ~~I have updated supported-technologies.asciidoc~~
• [ ] ~~Added an API method or config option? Document in which version this will be introduced~~

Related issues

• Closes #1509

[cla-checker-service[bot]]

💚 CLA has been signed

[gregawoods]

Hi - Let me know if there is anything I can do to help move this along. Ruby 3.4.2 just came out and it would be awesome to see the elastic-apm library having support for it. Thanks!

[netshade]

Elastic, it's been 57 days. Please respond on this thread so that those of us that depend on this gem either know you have intention to support or can make our own plans that allow us to upgrade while still using APM.

[netshade]

It is unfortunate that lack of response on this thread has to be reasonably interpreted as Elastic has no plans to merge any time soon, and Ruby users should likely start vendoring/forking @gregawoods PR in order to continue safely into the future.

[gregawoods]

I've merged in the latest changes from main and cleared up a small conflict. Would still love to see this accepted eventually, or really any formal acknowledgement in this space to show that it's on the road map.

[jclusso]

@colleenmcginnis @v1v @KOTungseth can someone respond to this from Elastic?

[gregawoods]

Ruby 3.4.3 was recently released. As more and more folks start adopting 3.4 I’m sure it would be appreciated to see some feedback here. 🙂

[estolfo]

@gregawoods Thanks a lot for this contribution! I'll work on getting a release out asap with ruby 3.4 support.

[estolfo]

Just a quick update @gregawoods: I need to update the tests so they pass on main and then we can rebase this so the tests can pass and it can be merged. I've opened a PR in which I'm working through the test failures. I hope it won't take too long :)

[gregawoods]

Understood, thank you so much!

[estolfo]

hey again @gregawoods I've fixed the tests and merged the changes into main. Would you mind rebasing so we can see if the tests pass on Ruby 3.4?

[gregawoods]

hey again @gregawoods I've fixed the tests and merged the changes into main. Would you mind rebasing so we can see if the tests pass on Ruby 3.4?

Done 👍

[gregawoods]

I've pushed up a handful of test fixes. Sidekiq changed their job wrapper class name, and ruby 3.4 dropped mutex_m from the standard library.

Mongo just has some changes to white space in it's output.

The failing rails 4 test may be more problematic. It looks like the correct thing to do may be to simply add an entry to exclude.yml. Edit: I added the exclusion, but happy to roll it back if that was the wrong call.

[gregawoods]

Taking a look at the latest round of failures. This one complains about a missing symbol in sqlite3. I now see that we should probably be excluding the ruby-3.4/rails-5 combination (e.g. ruby-3.1/5-2 is already excluded), so I'll add that.

The rest all look like this:

      #<Logger:0x19f9287a @level=1, @level_override={}, @logdev=nil, @default_formatter=#<Logger::Formatter:0x4924ad09 @datetime_format=nil>, @progname=nil, @formatter=nil> received :error with unexpected arguments
         expected: (/OpenSSL::SSL::SSLError/)
              got: ("[ElasticAPM] [THREAD:5116]: APM Server not responding in time, terminating request")

The test points to https://self-signed.badssl.com and expects a response.

let(:config) do
  Config.new(server_url: 'https://self-signed.badssl.com')
end

This feels to me like it could be related to the badssl.com service being temporarily flaky or returning bad responses. Do we think this requires some additional fixes here, or is this a "try it again" kind of situation?

[estolfo]

hey @gregawoods thank you so much for going through the test failures and adding some fixes and exclusions! I agree with excluding rails 5 from testing on ruby 3.4, that seems like an unlikely combo. I'll run the tests today and work through the remaining failures.

Update: the json specs are failing because this change in the json gem creates two spans when JSON#parse! is called, one for JSON.parse and one for JSON.parse! Opened this PR to update the json spy spec.

Another update: I've fixed the failing json spec and merged the change into main. Would you mind rebasing? Thanks!

[gregawoods]

@estolfo Latest changes have been rebased in. Thanks!

[estolfo]

I don't know at a first glance what's up with those jruby failures, as they seem to have just started with no related functional changes to the code...I'll have to dig into it and will hopefully have an update soon.

[estolfo]

It looks like the jruby failures are due to a bug, fixed here, but yet to be released. Update: it has been released, but maybe ci isn't picking up the latest version of the connection_pool gem. I'll run the tests again.

[estolfo]

run docs build

[estolfo]

run docs-build

[estolfo]

v4.8.0 is released with this contribution. Thanks again @gregawoods!

[gregawoods]

Awesome! Thanks for all your hard work fighting with the tests @estolfo.