Explicit use of `--source` should not fallback to other sources

[jacobbednarz]

This morning I encountered some unexpected behaviour however upon stewing on it for a bit and thinking about it further, I'm unsure if it's an issue or just something to be more conscious of. I thought I'd raise it here and get some additional views on it.

The scenario was that I was looking to install a gem from a private repository using gem install my_gem --source "https://my.private.gemserver.com" however unknown to me at the time, our private gem server was blocking traffic and silently failing requests. I wasn't paying a great deal of attention at the time and the end of the output showed that the gem I was after was successfully installed. A while later, I started debugging why a bunch of the expected functionality wasn't working and eventually I ran a bunch of checksums and found that this wasn't the intended gem.

There were a couple of contributing factors here that made this slightly more difficult to identify:

• Our private gem server was blocking traffic at the time and failing to respond with the required resources. This failure was silent and not apparent without the additional checksums or using using --vebose on the gem install.
• The gem I was looking to install was named the same as a publicly available gem.

Questions:

• Should calling gem install with --source be more restrictive on where it gets the requested source gem from? I understand the dependencies may need to come from other sources however would a "only try in this location for the gem I'm requesting" policy be useful? Or should this be an additional flag to provide with --source?

• Should rubygems fail loudly when the requested source is not returning the expected response and ends up falling back to another? Right now it just moves onto the next source. Here is an example of what I get if the private gem server 403's but is present in rubygems.org. Note: This isn't the gem version (or code) that I am expecting.

  $ gem install gem_name --source "https://my.private.gemserver.com/"
  Successfully installed gem_name-0.1.2
  1 gem installed
  

  gem sources -a makes it very obvious when a source is unavailable.

  $ gem source -a "https://my.private.geserver.com"
  Error fetching https://my.private.geserver.com: 
      bad response Forbidden 403 (https://my.private.geserver.com/specs.4.8.gz)
  

• Would it be worth while adding a --checksum flag that you could provide an expected checksum and should it mismatch, fail to complete the installation?

Looking forward to hearing your 2 cents on this!

[colby-swandale]

This is an excellent issue report and i agree with the argument that it should fail instead of falling back to other sources (this is technically a security issue as well).

[jacobbednarz]

Thanks @colby-swandale! I did initially think about throwing in security as a concern here however all the possible scenarios I came up with were highly unlikely and involved something else being compromised prior to rubygems being the cause so I didn't bother. If you have some scenarios in mind where you could see this being a realistic issue, I'm happy to explore those too.

[segiddins]

Just copy/pasting this from gem help install (not saying the way it is now is ideal):

        --clear-sources              Clear the gem sources
    -s, --source URL                 Append URL to list of remote gem sources

[duckinator]

I agree that it should clear the sources automatically if --source is specified.

I tried it locally, and it looks like (as @segiddins alluded to), passing --clear-sources before any --source/-s flags is a good workaround:

gem install my_gem --clear-sources --source "https://my.private.gemserver.com"

[jacobbednarz]

Thanks for the feedback @segiddins and @duckinator 🍭

--clear-sources --source was something that we tried but because our private gem has dependencies that are fetched from rubygems.org, it doesn't work and the installation fails. Reflecting on the flags I've used here, they are probably being misused however it does raise the question of how one should install a gem from a private gem server and be certain the requested gem came from that source but it's dependencies can come from anywhere?

[duckinator]

Oh, yeah, that does sound like it wouldn't work with the current behavior!

I'm not sure what a good approach to the command-line flags would be, but perhaps a Gemfile with a source block (as mentioned on https://bundler.io/gemfile.html ) and then using Bundler would work here?

Something along the lines of this:

source "https://rubygems.org"

source "https://my.private.gemserver.com" do
  gem "my_gem"
end

Unfortunately I have no way to test this, so I'm not sure if a source block affects dependencies.

[jacobbednarz]

Thanks for the suggestion! That seems closer to the behaviour that I would be expecting for that block with one minor issue.

Gemfile with similar gem setup where we have a private gem server with a commonly named gem that is also in rubygems.org.

source "https://rubygems.org"

source "https://my.private.gemserver.com" do
  gem "nokogiri"
end

When gemserver is unavailable

Fetching source index from https://my.private.gemserver.com/
Retrying fetcher due to error (2/4): Bundler::HTTPError Could not fetch specs from https://my.private.gemserver.com/
Retrying fetcher due to error (3/4): Bundler::HTTPError Could not fetch specs from https://my.private.gemserver.com/
Retrying fetcher due to error (4/4): Bundler::HTTPError Could not fetch specs from https://my.private.gemserver.com/
Could not fetch specs from https://my.private.gemserver.com/

When gemserver is online but the gem isn't in the intended repository.

Fetching gem metadata from https://my.private.gemserver.com/.
Retrying dependency api due to error (2/4): Bundler::MarshalError TypeError: incompatible marshal file format (can't be read)
  format version 4.8 required; 60.63 given
Retrying dependency api due to error (3/4): Bundler::MarshalError TypeError: incompatible marshal file format (can't be read)
  format version 4.8 required; 60.63 given
Retrying dependency api due to error (4/4): Bundler::MarshalError TypeError: incompatible marshal file format (can't be read)
  format version 4.8 required; 60.63 given


Fetching gem metadata from https://rubygems.org/..............
Fetching version metadata from https://rubygems.org/..
Could not find gem 'nokogiri' in rubygems repository https://my.private.gemserver.com/.
Source does not contain any versions of 'nokogiri'

But...if there are dependencies that aren't in the private gemserver (when online) but are in rubygems.org, it fails.

Fetching gem metadata from https://rubygems.org/..
Fetching version metadata from https://rubygems.org/.
Resolving dependencies...
Bundler could not find compatible versions for gem "gem_name":
  In Gemfile:
    gem_name was resolved to 0.1.0, which depends on
      httparty (~> 0.15)

Could not find gem 'httparty (~> 0.15)', which is required by gem 'gem_name', in any of the sources.

As a stop gap for this issue, we've implemented an external checksumming for the gem installation to ensure it is aligning with what version/gem we are expecting to see installed however I'm still unsure if this issue and the above sample is something we want to fix or it works well enough for most in it's current state.

[elondres-mim]

I agree that it should clear the sources automatically if --source is specified.

I tried it locally, and it looks like (as @segiddins alluded to), passing --clear-sources before any --source/-s flags is a good workaround:

gem install my_gem --clear-sources --source "https://my.private.gemserver.com"

Does this still work for others? When I run gem install --clear-sources -s <my-rubygems-server> the gem command just exits without any output.

[deivid-rodriguez]

Working fine for me!

As per this issue, I totally agree with it too.

As per how to do it, we could print a deprecation warning when --source is run without --clear-sources, and tell the user to set a new configuration clear_sources: true in their .gemrc file to silence the warning and enable the new behavior.

[deivid-rodriguez]

I was thinking about this, again, and I believe falling back to rubygems.org is still useful for indirect dependencies.

So, instead, I think we should change gem install my_gem --source "https://my.private.gemserver.com" to install my_gem from https://my.private.gemserver.com or fail if not possible, but keep the rubygems.org fallback for dependencies of my_gem.