Error downloading the agent while running the diagnose

[fblupi]

Describe the bug

I've installed appsginal on my Ruby on Rails application. When I run the diagnose locally (using a MacOS M1) it works. But when I repeat the process on production Ubuntu with x86_64 arch, it fails with the error attached in the section below.

I tried downloading manually the package from the URL in the logs with curl and it works fine. I've also decompressed the downloaded file and run the diagnose task with the env variables _APPSIGNAL_AGENT_PATH and APPSIGNAL_AGENT_PATH pointing the folder where I decompressed with the same issues. I've also moved the downloaded files to the ext directory of the gem. But the issue persists as it tries to download the agent. The agent I downloaded seems to work as it returns me the version where calling it with the --version argument.

The exception is raised on this block: https://github.com/appsignal/appsignal-ruby/blob/main/ext/base.rb#L135

I also tried with the latest published version of the appsignal gem without success.

I'm running out of ideas. Could you help me out if there's anything I can do from my side?

To Reproduce

Steps to reproduce the behavior:

• Using AppSignal for Ruby gem version 4.4.0, Ruby 3.1.7, and OpenSSL 3.0.2
• Specifying the APPSIGNAL_PUSH_API_KEY env variable with a correct API key
• Running RAILS_ENV=production bundle exec appsignal diagnose raise this error:

Could not download archive from any of our mirrors. Please make sure your network allows access to any of these mirrors.
Attempted to download the archive from the following urls:
- URL: https://d135dj0rjqvssy.cloudfront.net/0.35.29/appsignal-x86_64-linux-all-static.tar.gz
  Error: ArgumentError: wrong number of arguments (given 4, expected 0)
- URL: https://appsignal-agent-releases.global.ssl.fastly.net/0.35.29/appsignal-x86_64-linux-all-static.tar.gz
  Error: ArgumentError: wrong number of arguments (given 4, expected 0)

[unflxw]

Hi @fblupi,

I have tried to reproduce the issue using the latest version of the AppSignal gem (OpenSSL 3.0.15, Ruby 3.1.7, AppSignal for Ruby 4.5.13) but I have not been able to reproduce it. Could you provide a reproduction case, like a GitHub repository with instructions that reliably cause this issue?

[fblupi]

This is the repo: https://github.com/Platoniq/decidim-ub. You can use the deps/appsignal branch. As I mention in the issue, I cannot reproduce it locally with development environment. I can only reproduce in a ubuntu server with production environment.

[unflxw]

@fblupi I have been unable to reproduce the issue so far.

First, I am booting the ruby:3.1.7 Docker image, cloning the Platoniq/decidim-ub repository, switching to the deps/appsignal branch, and running bundle install. This installs AppSignal for Ruby 4.4.0.

Then, I run RAILS_ENV=production bundle exec appsignal diagnose. No error like the one in the issue description appears. The diagnose report works as expected.

Note that the agent is installed at bundle install time, when the dependency is installed, not when you run the appsignal binary.

[fblupi]

In my case it seems the agent is being downloaded while running the appsignal diagnose. I'm going to share the diagnosis report via email to [email protected]

[unflxw]

Thank you! Please mention this GitHub thread in the support conversation.

[unflxw]

Can you confirm whether you're running bundle install before attempting RAILS_ENV=production bundle exec appsignal diagnose? If you could also share your Ubuntu version, that would allow me to attempt to reproduce it with greater accuracy.

[fblupi]

Yes, I run it during the deployment. I use Ubuntu 22.04 and the kernel version is 5.15.0-136

[shairyar]

Support conversation on Intercom (internal link)

[backlog-helper[bot]]

• This issue has not had any activity in 14 days. Please provide a status update if it is still relevant. Closed it if it is no longer relevant. Or move it to another column if it's blocked or requires another look at it. - (More info)

---

This is a message from the daily scheduled checks.

New issue guide | Backlog management | Rules | Feedback

[fblupi]

I'm still waiting for a response

[unflxw]

Hi @fblupi, apologies for not providing a more definitive response. We're still unable to reproduce this on our end. It is quite likely that it relates to your server's network connection. Could it be that it's sitting behind a proxy that intercepts or blocks connections?

[rofiuddin]

Chiming in here, think I have reproduced the issue. It has something to with openssl 3.3.0 dependency which both OP and I have in common in Gemfile.lock. While running bundle install if installing openssl precedes appsignal installation, the extension fails to download agent and dumps the error report same as OP mentioned to ext/install.report. With openssl removed from gemset, it works fine and record success in install.report.

[unflxw]

Hi @rofiuddin, thank you for the additional information for this issue! We'll attempt to reproduce it with the openssl 3.3.0 dependency as described. Feel free to reach out over at [email protected] with any additional information.

[tombruijn]

I made a reproducible example in https://github.com/tombruijn/ruby_openssl_reproducible.

It seems the error is originating somewhere in the openssl gem and more specifically the C-extension. We can't fix that. I don't think there's anything wrong with how we call URI.open, which is what it fails on. (Upgrade to Ruby gem 4.5.16 to see the backtrace with the error.)

Edit: The best workaround is to upgrade RubyGems with gem update --system as described in https://github.com/appsignal/appsignal-ruby/issues/1414#issuecomment-3013953655

Another workaround that I've tested is downgrading the openssl gem to version 3.2.1, 3.1.1 or 3.0.3. Another workaround that works is upgrading to Ruby 3.2.8. If you can't specify the specific openssl version, this is your only method. Ruby 3.1.x is EOL, so it's good to upgrade anyway.

The openssl gem README states it has Ruby compatibility with Ruby 2.7+ for version 3.3.x of the gem, so this might be a bug in version 3.3.x of the gem.

[fblupi]

Thanks for the investigation. Unfortunately, I cannot easily upgrade the Ruby version. I upgraded the version of AppSignal to 4.5.16 and tried with the versions of openssl you mentioned (3.0.3, 3.1.1, and 3.2.1), but the error is still present.

- URL: https://appsignal-agent-releases.global.ssl.fastly.net/0.36.6/appsignal-x86_64-linux-all-static.tar.gz
  Error: ArgumentError: wrong number of arguments (given 4, expected 0)
/home/user/.rbenv/versions/3.1.7/lib/ruby/3.1.0/openssl/ssl.rb:277:in `initialize'
/home/user/.rbenv/versions/3.1.7/lib/ruby/3.1.0/openssl/ssl.rb:277:in `new'
/home/user/.rbenv/versions/3.1.7/lib/ruby/3.1.0/openssl/ssl.rb:277:in `decode'
/home/user/.rbenv/versions/3.1.7/lib/ruby/3.1.0/openssl/ssl.rb:277:in `block in verify_certificate_identity'
/home/user/.rbenv/versions/3.1.7/lib/ruby/3.1.0/openssl/ssl.rb:275:in `each'
/home/user/.rbenv/versions/3.1.7/lib/ruby/3.1.0/openssl/ssl.rb:275:in `verify_certificate_identity'
/home/user/.rbenv/versions/3.1.7/lib/ruby/3.1.0/net/protocol.rb:46:in `connect_nonblock'
/home/user/.rbenv/versions/3.1.7/lib/ruby/3.1.0/net/protocol.rb:46:in `ssl_socket_connect'
/home/user/.rbenv/versions/3.1.7/lib/ruby/3.1.0/net/http.rb:1089:in `connect'
/home/user/.rbenv/versions/3.1.7/lib/ruby/3.1.0/net/http.rb:995:in `do_start'
/home/user/.rbenv/versions/3.1.7/lib/ruby/3.1.0/net/http.rb:984:in `start'
/home/user/.rbenv/versions/3.1.7/lib/ruby/3.1.0/open-uri.rb:323:in `open_http'
/home/user/.rbenv/versions/3.1.7/lib/ruby/3.1.0/open-uri.rb:741:in `buffer_open'
/home/user/.rbenv/versions/3.1.7/lib/ruby/3.1.0/open-uri.rb:212:in `block in open_loop'
/home/user/.rbenv/versions/3.1.7/lib/ruby/3.1.0/open-uri.rb:210:in `catch'
/home/user/.rbenv/versions/3.1.7/lib/ruby/3.1.0/open-uri.rb:210:in `open_loop'
/home/user/.rbenv/versions/3.1.7/lib/ruby/3.1.0/open-uri.rb:151:in `open_uri'
/home/user/.rbenv/versions/3.1.7/lib/ruby/3.1.0/open-uri.rb:721:in `open'
/home/user/.rbenv/versions/3.1.7/lib/ruby/3.1.0/open-uri.rb:29:in `open'
/home/user/project/shared/vendor/bundle/ruby/3.1.0/gems/appsignal-4.5.16/ext/base.rb:143:in `block in download_archive'
/home/user/project/shared/vendor/bundle/ruby/3.1.0/gems/appsignal-4.5.16/ext/base.rb:131:in `each'
/home/user/project/shared/vendor/bundle/ruby/3.1.0/gems/appsignal-4.5.16/ext/base.rb:131:in `download_archive'
extconf.rb:22:in `install'
extconf.rb:93:in `<main>'

There I see we are not accessing the openssl gem. As it's using the code inside the own installation of Ruby.

[tombruijn]

We're seeing the same backtrace and yes, you can't see a openssl gem in the backtrace, but removing the openssl ruby gem from this scenario fixes it. You can't even really see the place it goes wrong, because it's in the C-extension from the openssl gem.

I remembered that we ran into a similar issue before, which was a bug in rubygems: https://github.com/rubygems/rubygems/issues/6611.

The problem there was that the wrong version of C-extension of the Psych gem (the Ruby YAML library) was loaded during bundle install. When a newer version of that gem was installed, the newer version of the C-extension was loaded for the gem version installed in the Ruby version. The same seems to be happening here.

The fix there was released in rubygems, and to get that fix you need to run gem update --system to update rubygems, before running bundle install. It works in the example setup I made on Ruby 3.1.7. Please try it and let us know if it fixes it for you.

[fblupi]

I updated rubygems and the bundler to the latest versions but I'm still facing the same issue. I guess I won't be able to use AppSignal until I make my application compatible with Ruby 3.2.x

[tombruijn]

That's really strange, none of these solutions/workarounds work for your app.

Do make sure it's run before the bundle install command is run in this Dockerfile. Ideally it's run before this line in the Dockerfile it inherits from Assuming you use the decidim/decidim:0.28.1 image specified in the app.

If even that doesn't work, the best way forward would probably be to upgrade Ruby.

[tombruijn]

@rofiuddin did running gem update --system fix it for you?

[rofiuddin]

@rofiuddin did running gem update --system fix it for you?

yes indeed @tombruijn, this fixed the issue along with upgrading appsignal gem to newer version 4.5.17. I guess the order here is important for the newer C-extension to get loaded.

[tombruijn]

I guess the order here is important for the newer C-extension to get loaded.

Yes, you have to make sure to update the rubygems versions before installing the appsignal version. If you already have the appsignal gem installed, upgrading rubygems won't do anything. You'd have to uninstall the appsignal gem first and then reinstall it.