guides
guides copied to clipboard
performance problems with RubyGems and Artifactory
we are seeing a massive performance hit when trying to do a bundle install where the source is an Artifactory based repository.
For a Gemfile where one gem changed some minor versions, 'bundle install' recalculation took about 5 seconds. Now it takes 11 minutes.
#292 seems like an explanation from Artifactory's side, but rubygems discounts this as a documentation issue for the compact API.
However, the compact API should be invoked by the latest version of bundler (2.4.13 at this writing) but apparently isn't. Instead two fallback apis are called resulting in 404s, which then trigger the third fallback requesting gemspecs for all gem versions (which takes about 11 minutes on my system).
This is an unacceptable change in performance that seems related to the changes rolled out in the compact API. We are urgently searching for solutions or workarounds to this issue.
references
- #292
- https://jfrog.atlassian.net/browse/RTFACT-13635
- https://jfrog.com/help/r/artifactory-rubygems-org-dependency-api-deprecation
- https://blog.rubygems.org/2023/02/22/dependency-api-deprecation.html
how the issue presents
- a call to
artifactory/api/gems/gems-repos/versions
fails with a 404. - a fallback to
artifactory/api/gems/gems-repos/api/v1/dependencies
fails with a 404. - finally, the third fallback initiates a download of all the gemspecs for all gem versions in order to resolve the dep tree locally
$ bundle install --verbose
Running `bundle install --verbose` with bundler 2.4.13
Found changes from the lockfile, re-resolving dependencies because the dependencies in your gemfile changed
HTTP GET [http://repository/artifactory/api/gems/gems-repo...](http://repository/artifactory/api/gems/gems-repos/versions)
HTTP 404 Not Found [http://repository/artifactory/api/gems/gems-repo...](http://repository/artifactory/api/gems/gems-repos/versions)
Bundler::Fetcher::FallbackError: Net::HTTPNotFound: [http://repository/artifactory/api/gems/gems-repo...](http://repository/artifactory/api/gems/gems-repos/versions)
/DONOTDELETE/shared/rvm/gems/ruby-3.1.2@app/gems/bundler-2.4.12/lib/bundler/fetcher/downloader.rb:45:in `fetch'
/DONOTDELETE/shared/rvm/gems/ruby-3.1.2@app/gems/bundler-2.4.12/lib/bundler/fetcher/compact_index.rb:124:in `call'
...
HTTP GET [http://repository/artifactory/api/gems/gems-repo...](http://repository/artifactory/api/gems/gems-repos/api/v1/dependencies)
HTTP 404 Not Found [http://repository/artifactory/api/gems/gems-repo...](http://repository/artifactory/api/gems/gems-repos/api/v1/dependencies)
...
Fetching source index from [http://repository/artifactory/api/gems/gems-repo...](http://repository/artifactory/api/gems/gems-repos/)
HTTP GET [http://repository/artifactory/api/gems/gems-repo...](http://repository/artifactory/api/gems/gems-repos/quick/Marshal.4.8/amq-protocol-0.8.3.gemspec.rz)
HTTP 200 OK [http://repository/artifactory/api/gems/gems-repo...](http://repository/artifactory/api/gems/gems-repos/quick/Marshal.4.8/amq-protocol-0.8.3.gemspec.rz)
...
# thousands more gemspec requests
environment
$ ruby -v
ruby 3.1.2p20 (2022-04-12 revision 4491bb740a) [x86_64-linux]
$ bundle --version
Bundler version 2.4.12
$ curl https://repository/artifactory/api/system/version
{
"version" : "7.46.11",
"revision" : "74611900",
"addons" : [ "ha", "build", "docker", "vagrant", "replication", "filestore", "plugins", "gems", "composer", "npm", "bower", "git-lfs", "nuget", "debian", "opkg", "rpm", "cocoapods", "conan", "vcs", "pypi", "release-bundle", "jf-connect", "jf-event", "replicator", "keys", "alpine", "cargo", "chef", "federated", "git", "observability", "pub", "rest", "swift", "conda", "terraform", "tracker", "license", "puppet", "ldap", "sso", "layouts", "properties", "search", "securityresourceaddon", "filtered-resources", "p2", "watch", "webstart", "support", "xray", "retention" ],
"license" : "0431f9d1fbae223e0bfa12c8ec1430fb0ca7ec1c3",
"entitlements" : {
"EVENT_BASED_PULL_REPLICATION" : true,
"SMART_REMOTE_TARGET_FOR_EDGE" : false,
"REPO_REPLICATION" : true,
"MULTIPUSH_REPLICATION" : true
}
}
I think this is the same problem being worked on at https://github.com/rubygems/rubygems/pull/6632?
I think this is the same problem being worked on at rubygems/rubygems#6632?
I think you might be right. I'll see if the PR fixes this issue on my side with a patch.
I don't think there is any way to patch this because of when it is loaded, so I'm waiting for the fix version I guess.
we are tracking this through the Artifactory side issue: https://jfrog.atlassian.net/browse/RTFACT-13635
Hey, just came across this. This is fixed in rubygems. Are you seeing the improvement?
I'm pretty sure this was indeed fixed, let's close! @coldnebo if you're still seeing issues please open a separate issue at https://github.com/rubygems/rubygems.