Run (package-refresh-contents) on first install each time.

[belak]

I'm sure there's a better way to do this, but this is what I have in my config right now for bootstrapping use-package and a few other packages. It ensures that the first package that gets installed will call refresh contents. I've had issues in the past with the specific version listed in the cache not existing, so that package fails to install. Refreshing the cache then installing that package works.

;; We want to track if we've run a refresh this time
(setq belak/did-refresh nil)

;; Small function to install a missing package
(defun package-ensure-installed (package)
  (unless (package-installed-p package)
    (unless belak/did-refresh
      (package-refresh-contents)
      (setq belak/did-refresh t))
    (package-install package)))

(package-ensure-installed 'use-package)

Currently, this is what is used:

(defun use-package-ensure-elpa (package &optional no-refresh)
  (if (package-installed-p package)
      t
    (if (or (assoc package package-archive-contents) no-refresh)
        (package-install package)
      (progn
        (package-refresh-contents)
        (use-package-ensure-elpa package t)))))

[thomasf]

I don't think we should work something overly complicated into this behaviour, much of package.el is hidden inside it's functions. If I have understood things correctly emacs25.1 will see quite alot of chages to package.el, maybe this should be solved there instead of in use-package?

[thomasf]

FYI, I just do a package-refresh-contents and restart emacs when I'm bootstrapping and that happens.. It's such an uncommon situation that it for me does not warrant special care, if you are boostrapping a new emacs installation every day you are probably doing something else wrong :)

[thomasf]

This is a part of my eary start up...

;;;; package.el
(eval-and-compile
  (setq
   package-enable-at-startup nil
   package-archives
   '(("melpa-stable" . "http://stable.melpa.org/packages/")
     ("melpa" . "http://melpa.org/packages/")
     ("marmalade"   . "http://marmalade-repo.org/packages/")
     ("org"         . "http://orgmode.org/elpa/")
     ("gnu"         . "http://elpa.gnu.org/packages/")
     ("sc"   . "http://joseito.republika.pl/sunrise-commander/")))

  (unless (boundp 'package-pinned-packages)
    (setq package-pinned-packages ()))

  (defun require-package (package &optional min-version no-refresh)
    "Install given PACKAGE, optionally requiring MIN-VERSION.
If NO-REFRESH is non-nil, the available package lists will not be
re-downloaded in order to locate PACKAGE."
    (if (package-installed-p package min-version)
        t
      (if (or (assoc package package-archive-contents) no-refresh)
          (package-install package)
        (progn
          (package-refresh-contents)
          (require-package package min-version t))))))

(defvar byte-compile-warnings nil)

(eval-when-compile
  (require 'package)
  (package-initialize t)
  (require-package 'use-package)
  (require 'use-package)
  ;; (require-package 'names)
  ;; (require 'names)
  (defmacro executable-find* (command)
    "Macro form of executable-find..."
    (executable-find command)))

I my init sequence this file is also important as it restricts packge.el to only be loaded at compile time:

https://github.com/thomasf/dotfiles-thomasf-emacs/blob/master/emacs.d/load-path.el

[colonelpanic8]

@thomasf This doesn't happen ONLY when you are bootstrapping a new emacs installation. It can also happen when you add new packages to your emacs configuration from another machine and then sync your init.el from that machine. This is something that I personally (and I assume many other emacs users) do quite often.

[thomasf]

If the package references by use-package is new the contents is refreshed. Last time I looked the package.el does not tell the caller why an package installation error occurred (the trigger would probably be on getting a http 404 response from package-install) so just assuming that it might have been a dependency availability error and retrying an installation if it fails falls in the too much of a hack category for my taste.

• If you want a hack I suggest you add it to your own init.el instead of having it inside of use-package which IMHO should not be guessing about error conditions.
• If you want it solved I suggest you fix it inside package.el.
• If it's already possible in emacs 25.1, I suggest a conditional check for packgage.el version which enables this feature for newer emacses

[belak]

Putting a package-refresh-contents inside init.el will only waste time most of the time. There's no point in running it if we're not installing new packages.

This actually almost never occurs during bootstrapping (at least if you ensure there are package-archive-contents, or something like that) and happens fairly often when adding a new package at runtime. Often times when I'm trying a new package, I'll make a use-package block in a scratch org file and use C-c C-c to run it, relying on use-package-always-ensure to install it. Having an old version of the package list causes this to break.

It is definitely a convenience thing, but I don't think it's that unreasonable of a workaround. Especially since there's currently no way to hook into use-package (that I know of) which would let users make their own workaround for this problem. If there was a way to provide my own function for ensuring a package was installed, that would be good enough.

[jeberger]

@thomasf

I don't think we should work something overly complicated into this behaviour

Note that @belak's solution is actually simpler than the implementation currently used by use-package.

If the package references by use-package is new the contents is refreshed.

No, the contents is refreshed only if the package is unknown. Installation will fail if an old version of the package appears in package.el's cache but the version on the server is newer. @belak's solution ensures that package-refresh-contents will be called at least once per session so that newer package versions can be found.

+1 for fixing this in use-package

[thomasf]

I was not saying that the suggested solutionis complicated, just that a proper solution which doesn't involve guessing about the remote package state might be complicated to implement.

Why is it better to add this inside use-package instead of package.el?

[vyp]

Sorry, I haven't followed this issue completely, but will this affect those that do not use any package repositories? And remember there are also use-package users who do not use package.el.

[jeberger]

@vyp No, it only affects people who use :ensure (directly or through use-package-always-ensure). People who don't use package.el will not be concerned by the existing behaviour nor by the proposed changes.

[jeberger]

@thomasf

a proper solution which doesn't involve guessing about the remote package state might be complicated to implement.

There is no guessing involved, updating the list of available packages before attempting to install or upgrade a package is simply best practice when dealing with a remote package repository.

Why is it better to add this inside use-package instead of package.el?

Better? I don't know. Faster? almost certainly given the size of the fix and their respective release cycles. Of course, implementing a quick fix in use-package does not preclude working for a more comprehensive solution in package.el...

[vyp]

@jeberger Thanks for clarifying.

[thomasf]

@jeberger alright!

I generally like better solutions solutions rather than getting stuff upstream as fast as possible.. In the end, it's all elisp so the fastest way will probably always be to add work arounds to ones own init.el if they are needed. I probably have a couple of thousand of lines in my init which only are there to fix or work around integration problems and fix some bugs.. It also works well..

Having said that I agree with the idea that doing a refresh when its needed isn't that bad of an idea. Maybe looking at the timestamps of the elpa/archivies/.../archive-contents files are a better way to identify if a refresh should be done and create a defcustom which sets the TTL for archive file renewals...

An example when the assumption to always update one time is totally wrong: I just have updated some packages and see a new package in the package list which I want to try out, I usually do that with use-package in the scratch buffer which with this automatic refreshing would immediately refresh the package contents even if I just did that before upgrading a minute earlier.

I still feel that this should go into package.el instead though.. especially since use-package also will be included in emacs in the future..

[phillord]

I'm using this solution to the problem.

(advice-add 'use-package-ensure-elpa
            :before
            (lexical-let ((done nil))
              (lambda (&args)
                (when (not done)
                  (message "Refreshing contents from use-package-ensure-elpa")
                  (package-refresh-contents)
                  (setq done t)))))

For myself, I think that this should go into use-package for the time being. I'd agree having a solution in package.el also would make sense. But, for the majority use of use-package, which is in the .emacs this works fine. And, for interactive use, worse case is that the package gets refreshed twice.

[belak]

That solution partially works, though it also will run a refresh on startup if no packages need to be installed...

After playing around with it, I've ended up with this, based on @phillord's initial work.

(setq belak-refreshed nil)
(advice-add 'use-package-ensure-elpa
            :around
            (lambda (orig-fun package &optional no-refresh)
              (if (package-installed-p package)
                  t
                (progn 
                  (when (not belak-refreshed)
                    (message "Refreshing contents from use-package-ensure-elpa")
                    (package-refresh-contents)
                    (setq belak-refreshed t))
                  (apply orig-fun package no-refresh)))))

I don't know elisp that well, so I'm open to improvements... one thing though, I don't have lexical-let for some reason, so I had to pull the variable out... looks like lexical-let is a part of the cl stuff included with emacs?

[phillord]

@belak Yes, am slightly embarrassed to note that you are entirely correct and I screwed up my code. It does refresh when any ":ensure" keyword is found.

I'm trying this out now:

(advice-add 'package-install
            :before
            (lexical-let ((done nil))
              (lambda (&args)
                (when (not done)
                  (message "Refreshing contents from package-install")
                  (package-refresh-contents)
                  (setq done t)))))

And, yes, lexical-let is part of cl.el. This does need fixing (or support anyway) in package.el. Has anyone submitted a bug report yet?

[raxod502]

I also would be glad to see this issue resolved. It is very unprofessional for use-package to simply fail sometimes, because it doesn't have an updated version of the package list.

For instance, in my configuration there is a configuration variable for whether you want to use Helm or Ivy. If you decide to switch from one to the other a week or two after setting things up for the first time, you will invariably get an error because one or more of the new packages have received updates and use-package can't find them (because it doesn't call package-refresh-contents).

For what it's worth, I think the following is a more elegant workaround (it does not require cl):

(defun my-package-install-refresh-contents (&rest args)
  (package-refresh-contents)
  (advice-remove 'package-install 'my-package-install-refresh-contents))

(advice-add 'package-install :before 'my-package-install-refresh-contents)

[belak]

I ended up just defining the variable globally so I don't need cl... https://github.com/belak/dotfiles/blob/master/emacs.d/README.org#package-setup

[jkaye2012]

Just want to be sure, this issue is closed but is actually still a problem? I know we can all just fix this in our init files, but it really feels like something that a utility as useful as use-package should really handle for the user.

[raxod502]

Indeed, I think @jwiegley closed this as part of his effort to move away from package.el stuff since he didn't feel like he could support it. Now that :ensure has been separated out of use-package core however, it might be a good time to add this hack in use-package-ensure.el. Somebody should provide a pull request. (I probably won't do it anytime soon, since I don't use package.el anymore.)

I've reopened the issue for now, but if @jwiegley feels that this is an inappropriate change to make in use-package-ensure.el, he can go ahead and close it again :)

[buhtz]

In my understanding package-refresh-content should be called automatic every time (the first time/only once) when installing a new package fails.