configuration-layer/update-packages doesn't handle system Emacs packages properly

[Atemu]

Description :octocat:

<<DESCRIPTION OF THE PROBLEM>>

Spacemacs tries to update a package that has been installed system-wide, fails and doesn't update the packages that come after it alphabetically.

Reproduction guide :beetle:

• Start Emacs with packages managed by the system (e.g. nix-shell -p '(emacsWithPackages [ emacsPackages.emacs-libvterm ])' --run emacs if you have Nix installed)
• Have a layer enabled which uses a system package (e.g. (shell :variables shell-default-shell 'vterm))
• Try to update Spacemacs' packages (M-x configuration-layer/update-packages RET)

Observed behaviour: :eyes: :broken_heart: <<DESCRIPTION OF THE OBSERVED BEHAVIOUR>>

(Spacemacs) --> preparing update of package vterm... [8/9]
package-delete: Package ‘vterm-20200107.1419’ is a system package, not deleting

(No further log, it just stalls and even begin to touch [9/9])

Expected behaviour: :heart: :smile: <<DESCRIPTION OF THE EXPECTED BEHAVIOUR>>

It shouldn't even try to update or delete vterm and/or just continue with the next package.

System Info :computer:

• OS: gnu/linux
• Emacs: 26.3
• Spacemacs: 0.300.0
• Spacemacs branch: develop (rev. 015303a70)
• Graphic display: t
• Distribution: spacemacs
• Editing style: hybrid
• Completion: ivy
• Layers:

(rust docker emacs-lisp git
      (ibuffer :variables ibuffer-group-buffers-by 'projects)
      (ivy :variables ivy-enable-advanced-buffer-information t)
      latex multiple-cursors nixos org pdf
      (shell :variables shell-default-shell 'vterm)
      theming treemacs)

• System configuration features: XPM JPEG TIFF GIF PNG RSVG SOUND DBUS GSETTINGS GLIB NOTIFY LIBSELINUX GNUTLS LIBXML2 FREETYPE M17N_FLT LIBOTF XFT ZLIB TOOLKIT_SCROLL_BARS GTK3 X11 XDBE XIM MODULES THREADS LIBSYSTEMD

Backtrace :paw_prints:

<<BACKTRACE IF RELEVANT>>

[emacs18]

Normally packages installed by users are installed under the directory specified via package-user-dir variable. Emacs searches this directory when looking for installed packages. In addition it also searches under the directories specified in package-directory-list.

I think the problem you are reporting is that

• some package is installed under one of the directories listed in package-directory-list which is not same as package-user-dir.
• spacemacs tries to install a newer version of that package and so first tries to delete the older version
• however it fails to delete, because package-delete function refuses to delete packages from any directory other than package-user-dir.

I can think of at least three ways to solve this problem.

• Manually delete the package if you are allowed to do so, i.e., you have access permissions to remove files
• Set package-user-dir to be the same as the directory with the package to be removed. Of course this assumes that you have write access to that directory and that no one else uses that directory.
• Set package-directory-list to nil so that spacemacs will look only in the directory where you have full access to so that packages can be removed. I think this is the safest solution.

This problem is not related to spacemacs. It seems to be the way built-in package.el was designed to work. Thus if you would like this to be changed, then bug report should be submitted to emacs itself rather than to spacemacs.

[Atemu]

The problem is that I need spacemacs to load packages from "the system" and not try to update or replace them with any other version. I want it to load the slightly outdated version installed in "the system".

The reason is that libvterm can't be compiled manually on Nix systems due to https://github.com/akermu/emacs-libvterm/issues/115, so I need Spacemacs to always use the Nix-made vterm pkg.

Emacs does differentiate packages installed in "the system" in the status field (they are "external") and since Spacemacs provides both the update function and a variable to exclude packages, I thought this might be a configuration issue rather than an Emacs bug.

I actually (sort of) already fixed this in my personal dotspacemacs (see commit above your comment).

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Please let us know if this issue is still valid!

[lebensterben]

As described above

The reason is that libvterm can't be compiled manually on Nix systems due to akermu/emacs-libvterm#115

The direct solution to the issue is to fix the problem in upstream. So this has nothing to do with Spacemacs.

Closing this since the upstream issue is closed.

[Atemu]

The problem has nothing to do with libvterm.

The problem is that spacemacs tries* to install and update packages that are already provided by the system. It just happened to be libvterm in my case because I had to use install it from the system due to that bug but it could be any package.

I would actually like to migrate to a setup where Spacemacs' package management would only be responsible for ad-hoc installation and trial with everything else being provided by the "system". If this bug still exists*, it'd block that endavour.

* Note: I haven't tested this behaviour recently

[lebensterben]

@Atemu In general I don't think a system package manager should distribute Emacs packages. They strictly should be handled by Emacs itself.

There are discussions about adding better support to nix OS or package manager. But it's not getting momentum now.

[Atemu]

For most system PMs that would be my opinion aswell but Nix is a very different story; there it's the complete opposite.

Even the creator of use-package uses Nix for their Emacs package management nowadays.

Nix support is way outside the scope of this issue of course; all I can ask for is Spacemacs' configuration-layer/update-packages to not crash and burn when there happen to be packages provided by the system as it did when I opened the issue.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Please let us know if this issue is still valid!

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Please let us know if this issue is still valid!

[Atemu]

I tried to reproduce this using:

nix-shell -p '(emacsWithPackages [ emacsPackages.vterm emacsPackages.pdf-tools ])' --run emacs

and this is what happened:

Updating package archives, please wait...
--> refreshing package archive: nongnu... [3/3]
--> Found 2 package(s) to update:
pdf-tools (won't be updated because package is frozen)
vterm (won't be updated because package is frozen)
--> performing backup of package(s) to update...

--> 0 package(s) to be updated.

Restart Emacs to install the updated packages. (SPC q r)

In the interactive prompts it still asks me whether I want to update these packages but apparently filters them out later.

Restarting does not do anything as expected and the local packages are not installed through other means.

Apart from the minor cosmetic issue of it still prompting me, I'd consider this issue resolved.