gem update installs system-wide updates in home

[exploide]

This is a follow up of #2273 which was closed due to inactivity.

When gem update is invoked with --user-install option (e.g. set in .gemrc), it installs updates to the home directory, as expected. But it also installs updates for system-wide gems there. Like the reporter of #2273, I also think this is usually not wanted.

I propose to only update user-installed gems when gem update is invoked with --user-install.

Alternatively, I'm interested in any other way to update exclusively user-installed gems as a workaround when this gets rejected.

This issue is related to:

• [ ] Network problems
• [ ] Installing a library
• [ ] Publishing a library
• [x] The command line gem
• [ ] Other

Here are my current environment details:

> gem env
RubyGems Environment:
  - RUBYGEMS VERSION: 3.0.3
  - RUBY VERSION: 2.6.3 (2019-04-16 patchlevel 62) [x86_64-linux]
  - INSTALLATION DIRECTORY: /usr/share/gems
  - USER INSTALLATION DIRECTORY: /home/jannik/.gem/ruby
  - RUBY EXECUTABLE: /usr/bin/ruby
  - GIT EXECUTABLE: /usr/bin/git
  - EXECUTABLE DIRECTORY: /usr/bin
  - SPEC CACHE DIRECTORY: /home/jannik/.gem/specs
  - SYSTEM CONFIGURATION DIRECTORY: /etc
  - RUBYGEMS PLATFORMS:
    - ruby
    - x86_64-linux
  - GEM PATHS:
     - /usr/share/gems
     - /home/jannik/.gem/ruby
     - /usr/local/share/gems
  - GEM CONFIGURATION:
     - :update_sources => true
     - :verbose => true
     - :backtrace => false
     - :bulk_threshold => 1000
     - "gem" => "--no-document --user-install --bindir /home/jannik/.gem/bin"
  - REMOTE SOURCES:
     - https://rubygems.org/
  - SHELL PATH:
     - /home/jannik/.gem/bin
     - /home/jannik/.local/bin
     - /usr/local/sbin
     - /usr/sbin
     - /usr/local/bin
     - /usr/bin
     - /bin

I will abide by the code of conduct.

[bronzdoc]

@exploide can you provide the contents of your .gemrc or how are you setting the --user-install ?

I can't reproduce the issue, either using gem: --user-install or update: --user-install confifured in my .gemrc

[exploide]

Thanks for your reply @bronzdoc.

I'm surprised this is not reproducible on your end, but I will provide the necessary information below. The issue appears on multiple Fedora 30 machines for me.

# this is my gem and ruby
> gem --version
3.0.3
> ruby --version
ruby 2.6.3p62 (2019-04-16 revision 67580) [x86_64-linux]

# my .gemrc as requested
> cat .gemrc
gem: --no-document --user-install --bindir /home/jannik/.gem/bin

# let's delete all user-installed gems
> rm -rf .gem

# and see that gem still finds (system-wide) stuff to update
> gem update
Updating installed gems
Updating bigdecimal
Fetching bigdecimal-1.4.4.gem
Building native extensions. This could take a while...
Successfully installed bigdecimal-1.4.4
Updating csv
Fetching csv-3.1.1.gem
Successfully installed csv-3.1.1
Updating fileutils
Fetching fileutils-1.2.0.gem
Successfully installed fileutils-1.2.0
Updating io-console
Fetching io-console-0.4.8.gem
Building native extensions. This could take a while...
Successfully installed io-console-0.4.8
Updating logger
Fetching logger-1.4.1.gem
Successfully installed logger-1.4.1
Updating rdoc
Fetching rdoc-6.1.1.gem
Successfully installed rdoc-6.1.1
Updating rexml
Fetching rexml-3.2.2.gem
Successfully installed rexml-3.2.2
Updating rss
Fetching rss-0.2.8.gem
Successfully installed rss-0.2.8
Gems updated: bigdecimal csv fileutils io-console logger rdoc rexml rss

Let me know if you need further information.

[bronzdoc]

@exploide Ah! I tried with a specific gem 😬 definitely can reproduce using just gem update

Will give it a look, thanks for the information!

[bronzdoc]

Working on improving this @exploide. I agree gem update --user-install should update only user installed gems.

[exploide]

Great news, thank you.

[deivid-rodriguez]

#2901 partially fixed this by not updating regular gem update into home. However, it still updates default gems into home, so I'll reopen this since the issue is not fully fixed yet.

[deivid-rodriguez]

What happened here? :thinking:

[hsbt]

@deivid-rodriguez I prepared and pushed rubygems-server privately. It may contained Fixed feature in its commit comment.

[deivid-rodriguez]

Aahm, makes sense. Thanks @hsbt!

[voxik]

I propose to only update user-installed gems when gem update is invoked with --user-install

I honestly don't understand the proposal. Why? If there is --user-install specified and e.g. json update is available, it should be installed into user directory. There is nothing surprising about this behavior IMHO. What could be surprising are the permission issues as mentioned in the #2273, but that would be different topic.

I'd also like to note that #2901 is non-systematic and breaks other stuff as reported in #6429 and I'd actually like to see it reverted for the moment.

[exploide]

I think in the end, our requirements aren't completely different. You don't want to install a package into user home when it's not really necessary (#6429). I don't want to install (updated) packages into user home when they are not really necessary (i.e. not dependencies for a tool I installed deliberately into user home).

My use case is the following: There are some system-wide ruby packages installed via package management, because they are required by something also installed system-wide. This is fine but I don't really care about these packages. They are there, they work, they are handled by distribution's package management. Then there are some ruby tools not included in the distribution's package management. I install them with gem install --user-install (the option actually comes from .gemrc). And I need to somehow keep them up to date. So I included gem update --user-install (followed by gem cleanup ) in my daily update routine (just like I do it with dnf upgrade, flatpak update and so on). Now it also updates all the system-wide installed (default) gems. But I don't need them in my home.

That's why I proposed this behavioral change.

Even if I would ignore the bloat, it annoys me that I always get the following warning in the output:

$ gem update --user-install
WARN: Unresolved or ambiguous specs during Gem::Specification.reset:
      psych (>= 4.0.0)
      Available/installed versions of this gem:
      - 5.1.0
      - 4.0.4
WARN: Clearing out unresolved specs. Try 'gem cleanup <gem>'
Please report a bug if this causes problems.
Updating installed gems
Nothing to update

psych 4.0.4 is installed system-wide so it cannot clean it up. That's fine. It also updated it to 5.1.0 in my home. I don't need that but fine. But now it also complains about this obvious cleanup issue. This situation cannot be resolved, no need to tell me.

So that's where I come from. But to be honest I'm not a big ruby user. I installed only two or three tools this way. If you all say "no that's intended" then close this as won't fix. But since two of the maintainers started addressing this issue, I thought it's reasonable.

[voxik]

The situation is not as easy as having some system wide or user installed packages. In reality, there is something we can call GEM_PATH and GEM_HOME (leaving default gems out of the discussion).

GEM_PATH refers to multiple locations and looks like this on my system:

  - GEM PATHS:
     - /usr/share/gems
     - /home/vondruch/.gem/ruby
     - /usr/local/share/gems

It could contain more locations and all the location are always used to resolve the dependencies and also for loading all gems for runtime.

The GEM_HOME is the default installation location, which is used for gem install or gem update. This is how it looks on my system:

  - INSTALLATION DIRECTORY: /usr/share/gems

Please note that the GEM_HOME is just one of the locations referenced in GEM_PATH and it can easily be modified:

$ GEM_HOME=/foo gem env | grep '\- INSTALL'
  - INSTALLATION DIRECTORY: /foo

And there is also --user-install option. But that is just convenient way of changing GEM_HOME.

As you can see, in this concept it is never anything like "do something with some specific repository". RubyGems always considers the GEM_PATH and if needed, it does modification into GEM_HOME.

Even if I would ignore the bloat, it annoys me that I always get the following warning in the output:

$ gem update --user-install
WARN: Unresolved or ambiguous specs during Gem::Specification.reset:
     psych (>= 4.0.0)
     Available/installed versions of this gem:
     - 5.1.0
     - 4.0.4
WARN: Clearing out unresolved specs. Try 'gem cleanup <gem>'
Please report a bug if this causes problems.
Updating installed gems
Nothing to update

I have never seen this, have no idea what does it mean and how to reproduce, but that seems something which should be addressed.

[voxik]

I don't want to install (updated) packages into user home when they are not really necessary (i.e. not dependencies for a tool I installed deliberately into user home).

BTW have you considered situation having system wide installed JSON of version e.g. 1.2.0 and updating user installed package FOO, which (newly) depends on JSON ~> 2. You might end up with updated JSON in your user directory (which goes against your request) or in aborted update.

[exploide]

In reality, there is something we can call GEM_PATH and GEM_HOME [...] It could contain more locations and all the location are always used to resolve the dependencies and also for loading all gems for runtime.

Yes I'm aware of that but this doesn't change my use case. But thanks anyway for summarizing this since I'm not that deep into the ruby ecosystem.

I have never seen this, have no idea what does it mean and how to reproduce, but that seems something which should be addressed.

If psych is installed system-wide and you run gem update with a .gemrc of gem: --no-document --user-install --bindir /home/user/.gem/bin this message should occur at latest on the second run of gem update. This is because it installs a newer version of psych into home and of course fails to uninstall the system-wide version (it should not try to do that I guess).

BTW have you considered situation having system wide installed JSON of version e.g. 1.2.0 and updating user installed package FOO, which (newly) depends on JSON ~> 2. You might end up with updated JSON in your user directory (which goes against your request) or in aborted update.

This wouldn't be against my request. It's totally fine (and expected) when dependencies of user-installed packages also get user-installed.

I know it's a bit complex with the search paths and the decision what should be considered in which case. If it's not desired to handle this as I proposed, it's fine for me to reject this. In the end I just need a way to deal with ruby tools that end up being user-installed and how to keep them up to date. Updating and duplicating everything from the system installation in my home felt bloated and unnecessary. That's why I proposed this. And then there are these annoying warnings I mentioned.

[voxik]

Trying to reproduce your issue, one of the ironies of #2901 is despite having Psych 4.0.4 installed system wide, the gem update psych fails :man_facepalming:

$ gem update psych
Updating installed gems
Nothing to update
Gems not currently installed: psych

Going to continue with gem install ...

[jlmuir]

I too have this issue and would love it if it were fixed! To summarize where things are, the issue is that there's no way to update just the gems you've installed as user-installed gems without explicitly listing them by name. That is,

$ gem update --user-install

updates gems that were not installed via

$ gem install <gem-name> --user-install

The use case is that I install gems with

$ gem install <gem-name> --user-install

and want to keep these user-installed gems up to date, so I run

$ gem update --user-install

I do not want that to update default gems nor bundled gems (as defined in Ruby Standard Gems) because those are provided with Ruby and are updated in a different way (e.g., via the OS's update mechanism if provided with the OS or via a package manager).

According to deivid-rodriguez's comment on Oct 31, 2019, PR #2901 only prevents bundled gems (I'm guessing he meant "bundled" gems when he said "regular" gems) from being updated, not default gems. So, this issue is only partially fixed.

Since listing the gems to update by name seems to work, a workaround would be to list the user-installed gems, and then run

$ gem update <gem-name> --user-install

for each of them, but gem seems to provide no way to list only user-installed gems as far as I can tell.

The workaround I'm left with is to separately maintain a list of the gems I've installed as user-installed gems (one gem name per line), and then, for example, if that list is at ~/.user-installed-gems, do

$ for each in $(cat ~/.user-installed-gems); do gem update $each --user-install; done

And that's sad.

[forthrin]

Arriving late to the party. Would like to know the proper way to rid this warning.

• gem cleanup has complained about this forever
• All gems are in /opt/homebrew (no user installs)
• Ironic that gem cleanup suggests to try gem cleanup
• macOS Sonoma

$ gem cleanup
Cleaning up installed gems...
WARN: Unresolved or ambiguous specs during Gem::Specification.reset:
      psych (>= 4.0.0)
      Available/installed versions of this gem:
      - 5.1.1.1
      - 5.0.1
WARN: Clearing out unresolved specs. Try 'gem cleanup <gem>'
Please report a bug if this causes problems.
Clean up complete

$ gem dependency $(gem list)     # note: irrelevant lines removed 
Gem mime-types-data-3.2023.1205
  psych (~> 3.0, development)

Gem psych-5.0.1
  stringio (>= 0)

Gem psych-5.1.1.1
  stringio (>= 0)

Gem rdoc-6.5.0
  psych (>= 4.0.0)

Gem rdoc-6.6.1
  psych (>= 4.0.0)

[thomaswitt]

I did gem uninstall psych stringio && gem update --user-install