not clear how to handle config which involves multiple packages

[aspiers]

Hey John :) Firstly, awesome work here. I saw you demo this at emacsconf but only just got round to trying it out. Based on the docs, it seems to cover exactly what I want (the focus on startup speed is especially awesome!), except for one area I am unclear on:

If I have some config which involves integration between multiple packages, how can I use use-package to ensure that config is run only when all those packages are loaded?

For example, guide-key and org-mode, or guide-key and key-chord, or key-chord and any mode which I want key chord bindings for. I'm sure you can easily think of many other combinations :)

If use-package can already do this then I guess this bug is a friendly request to make that more obvious in the README.md, and if it can't, please consider this a feature request :)

[jwiegley]

Interestingly enough, I just finish discussing this issue with someone else.

Right now use-package depends on the linear ordering of declarations within the file. If you want to make sure that a declaration is ignored if another package failed to load, use the :if directive to check for some featurep or some such that indicates whether the other package successfully loaded.

[aspiers]

Ah, but that would prevent one package loading if the other failed to load, and I don't want that...

[jwiegley]

Right, then in that case there is no way to say "Don't load this until X has loaded."

[aspiers]

I see - does closing this issue mean you wouldn't accept pull requests to implement this feature, or just that you don't intend to implement it yourself?

[jwiegley]

I would rather have this feature implemented by an extension package, like use-package-deps or something, rather than in the core. Unless you can show me that (a) it doesn't affect load speed at all, and (b) doesn't introduce excessive levels of complexity in the core code.

[aspiers]

I'd be very surprised if it would affect load speed or introduce complexity. Maybe worth considering what I would want this to look like ... here's an example:

(with-packages (org-mode guide-key)
  (defun guide-key/my-hook-function-for-org-mode ()
    (guide-key/add-local-guide-key-sequence "C-c")
    (guide-key/add-local-guide-key-sequence "C-c C-x")
    (guide-key/add-local-highlight-command-regexp "org-"))
  (add-hook 'org-mode-hook 'guide-key/my-hook-function-for-org-mode))

Then the code would only get run after the :config sections of both packages had been run. Achieving that is presumably pretty easy, but maybe some thought required to make sure it would compile cleanly?

[jwiegley]

I see what you mean, this is different from what I thought you originally meant. We could have a list of "post install" functions which return either nil or t to indicate whether they were able to run, and if they return t then we remove them from the list.

[aspiers]

Oh right :) What did you think I meant?

Yes, that list sounds good. I expect most of the predicates would be of the form (every 'featurep '(org-mode guide-key)), but it might make sense to support arbitrary predicates too.

[aspiers]

I just ran into another two use cases for this:

• (smartrep-define-key global-map "C-S-SPC" ...lots of multiple-cursors bindings...) when both smartrep and multiple-cursors are loaded
• (define-key region-bindings-mode-map "n" 'mc/mark-next-like-this) and similar when both region-bindings-mode and multiple-cursors are loaded

[Silex]

Just wanted to mention https://github.com/edvorg/req-package

[aspiers]

@Silex Thanks a lot, I'll take a look!

[aspiers]

AFAICS this is not really resolved until https://github.com/edvorg/req-package/issues/18 is resolved.

[marcinant]

Personally I solved this by running 'use-package' multiple times on the same package, just with different :if conditions.

[aspiers]

@marcinant Please could you provide an example of that? Even though req-package has supposedly resolved this, I can't find any clear documentation of how it's supposed to work.

[aspiers]

@jwiegley Any chance we could revisit this? req-package has not been updated in 2.5 years, and anyway I get the impression that more "recent" enhancements in use-package such as :after have partially or even fully removed the need for it. But it's still not clear to me how to handle config involving multiple packages. I think there is more need for this than ever, due to the explosion of emacs packages in the last few years which complement each other (e.g. treemacs, ivy, projectile to name a few).

Thanks!

[aspiers]

@marcinant's approach seems flawed, because if the :if condition fails, the config will never get run, but it should get run at the point when all relevant packages are loaded.

I am wondering if a variant of this approach would work, e.g. using my previous example:

(use-package org-mode
  :after guide-key
  :config
  (defun guide-key/my-hook-function-for-org-mode ()
    (guide-key/add-local-guide-key-sequence "C-c")
    (guide-key/add-local-guide-key-sequence "C-c C-x")
    (guide-key/add-local-highlight-command-regexp "org-"))
  (add-hook 'org-mode-hook 'guide-key/my-hook-function-for-org-mode))

But it's not clear whether this would interfere with any other (use-package org-mode ...) stanzas, e.g. preventing any of them from running until guide-key is loaded, which would certainly be undesirable. I still suspect that my previous proposal of a new with-packages macro would be the most user-friendly way to support this.

[aspiers]

Further issues with relying on req-package for this:

• The lack of maintenance means that even though it wraps around req-package, not all features of use-package work (I found this recently, although can't remember off-hand which ones don't). That means users have to mix and match usage of use-package and req-package, which generally feels wrong and makes debugging startup issues with either harder.

• req-package provides this functionality via :require, whereas use-package provides the keyword :requires for something slightly different. That's a confusing situation to be in ;-)

[conao3]

I don't read this whole thread, but use-package is not intended to constitute a nested use-package. This is a consistent view of @jwiegley.

FYI, leaf.el intended to this situation and it have :require keyword.

[aspiers]

Thanks for the reply.

I don't read this whole thread, but use-package is not intended to constitute a nested use-package. This is a consistent view of @jwiegley.

I'm not sure what exactly you mean by "nested", but I don't want to nest anything. I just want to be able to specify config which is automatically activated when a given list of packages are fully loaded.

However I have done some more experiments and I think I have figured it out. It seems that the trick is to include :defer t for the use-package stanza with the combined config. Here is an example, with the combined config in the middle of the three use-package calls:

(package-initialize)

(require 'use-package)
(setq use-package-verbose 'debug)

(use-package org
  :config
  (message ":config for just org")
  :commands org-mode)

(use-package org
  :after ivy
  :defer t
  :config
  (message ":config for org+ivy"))

(use-package ivy
  :config
  (message ":config for just ivy")
  :commands ivy-mode)

(message "Running (ivy-mode)")
(ivy-mode)
(message "------------------------------------")
(message "Running (org-mode)")
(org-mode)

If I run this via emacs -Q --batch -l ~/.emacs.d/test/combined-config.el, I see:

Running (ivy-mode)
Configuring package ivy...
   :config for just ivy
Configuring package ivy...done
------------------------------------
Running (org-mode)
Configuring package org...
   :config for just org
Configuring package org...done
Configuring package org...
   :config for org+ivy
Configuring package org...done

This is the exact desired behaviour - nothing is loaded until it is asked for, and the combined config is only triggered after both packages have loaded.

There is one imperfection: if I swap the order in which ivy-mode and org-mode are called, I see:

Running (org-mode)
Configuring package org...
   :config for just org
Configuring package org...done
------------------------------------
Running (ivy-mode)
Configuring package org...
   :config for org+ivy
Configuring package org...done
Configuring package ivy...
   :config for just ivy
Configuring package ivy...done

In the above, I would expect "just ivy" to appear before "org+ivy", not after. Another experiment revealed that this order is due to the org+ivy use-package being declared before the ivy one; swapping them around fixes the order, but this is not always an option when config is spread over multiple init files.

FYI, leaf.el intended to this situation and it have :require keyword.

That's interesting, and it looks like an impressive piece of work. It would be great if the README.md gave a comprehensive explanation of why you decided to write something from scratch rather than just working with the rest of the community to improve use-package. The current two-sentence explanation is not enough to understand this.

Anyway, given my discoveries, there are two remaining tasks to consider:

1. Document this approach in use-package's README.md.
2. Maybe add a (defmacro with-packages ...) which expands to (use-package a :after b :defer t ...), although I'm not sure how this would work with supporting things like :after (:any (:all foo bar) (:all baz quux)).

[conao3]

Oops, I should be looking at everything in the thread. You want to have fine control over the configuration loading order.

I might be able to solve this with leaf, but it's too off topic. Let's talk more about it in the leaf issue tracker. Anyway, there is a package similar to use-package, like req-package I just wanted to let you know that.

[jwiegley]

@aspiers I'm very interested in fixing the "order of declaration" problem, because using :after should not introduce a dependency on the order in which they're encountered. Do you have perhaps a small test that fails which shouldn't?

[aspiers]

Hi @jwiegley, thanks for the reply. Yes, if you use the above test case but swap the (ivy-mode) call with (org-mode), you'll see the issue I describe.

[aspiers]

To be clear, (use-package org :after ivy...) config is executed before (use-package ivy) config is if and only if its declaration is encountered before ivy's.

[conao3]

This problem is easily solved by a nested use-package. However, as I said before, this usage is not recommended for use-package. Also, since the use-package automatically expands the require, It should be noted that the :no-require keyword is required. Since your script couldn't run in a single file, I need to add the :ensure keyword.

;; ~/.debug.emacs.d/use-package/init.el

;; you can run like 'emacs -q -l ~/.debug.emacs.d/{{pkg}}/init.el'

(prog1 "prepare use-package"
  (prog1 "package"
    (custom-set-variables
     '(package-archives '(("org"   . "https://orgmode.org/elpa/")
                          ("melpa" . "https://melpa.org/packages/")
                          ("gnu"   . "https://elpa.gnu.org/packages/"))))
    (package-initialize))

  (prog1 "use-package"
    (unless (package-installed-p 'use-package)
      (package-refresh-contents)
      (package-install 'use-package))))

(require 'use-package)
(setq use-package-verbose 'debug)

(use-package org
  :defer t
  :config
  (message ":config for just org")
  (use-package *org-ivy-integration
    :no-require t
    :after ivy
    :config
    (message ":config for org+ivy")))

(use-package ivy
  :ensure t
  :defer t
  :config
  (message ":config for just ivy"))

(message "Running (ivy-mode)")
(ivy-mode)
(message "------------------------------------")
(message "Running (org-mode)")
(org-mode)

;; (message "Running (org-mode)")
;; (org-mode)
;; (message "------------------------------------")
;; (message "Running (ivy-mode)")
;; (ivy-mode)

Yes, two of the invocation orders produce the your intended result.

$ emacs -Q --batch -l ~/.debug.emacs.d/use-package/init.el
Running (ivy-mode)
Configuring package ivy...
:config for just ivy
Configuring package ivy...done
------------------------------------
Running (org-mode)
Configuring package org...
:config for just org
Configuring package *org-ivy-integration...
:config for org+ivy
Configuring package *org-ivy-integration...done
Configuring package org...done

$ emacs -Q --batch -l ~/.debug.emacs.d/use-package/init.el
Running (org-mode)
Configuring package org...
:config for just org
Configuring package org...done
------------------------------------
Running (ivy-mode)
Configuring package ivy...
:config for just ivy
Configuring package ivy...done
Configuring package *org-ivy-integration...
:config for org+ivy
Configuring package *org-ivy-integration...done

You can use leaf to produce the same result, but omit some keywords. FYI.

;; ~/.debug.emacs.d/leaf-use-package/init.el

;; you can run like 'emacs -q -l ~/.debug.emacs.d/{{pkg}}/init.el'
(when load-file-name
  (setq user-emacs-directory
        (expand-file-name (file-name-directory load-file-name))))

(prog1 "prepare leaf"
  (prog1 "package"
    (custom-set-variables
     '(package-archives '(("org"   . "https://orgmode.org/elpa/")
                          ("melpa" . "https://melpa.org/packages/")
                          ("gnu"   . "https://elpa.gnu.org/packages/"))))
    (package-initialize))

  (prog1 "leaf"
    (unless (package-installed-p 'leaf)
      (package-refresh-contents)
      (package-install 'leaf))))

(leaf org
  :config
  (message ":config for just org")
  (leaf *org-ivy-integration
    :after ivy
    :config
    (message ":config for org+ivy")))

(leaf ivy
  :ensure t
  :config
  (message ":config for just ivy"))

(message "Running (ivy-mode)")
(ivy-mode)
(message "------------------------------------")
(message "Running (org-mode)")
(org-mode)

;; (message "Running (org-mode)")
;; (org-mode)
;; (message "------------------------------------")
;; (message "Running (ivy-mode)")
;; (ivy-mode)

[aspiers]

Very interesting - thanks! IMHO it should be possible to do this without nesting, or relying on any particular ordering of the use-package declarations. Your :no-require t approach gave me an idea to try something like:

(use-package org
  :config
  (message "   :config for just org")
  :commands org-mode)

(use-package *org+ivy
  :no-require t
  :after (org ivy)
  :config
  (message "   :config for org+ivy"))

(use-package ivy
  :config
  (message "   :config for just ivy")
  :commands ivy-mode)

If that had worked, then it would probably be quite easy to write a with-packages macro to generate the combined declaration.

But unfortunately the :config for *org+ivy still gets run before the :config for ivy. Adding :defer t to it simply prevented *org+ivy from running at all.

So I wonder if this is some kind of subtle bug in when use-package chooses to run :config for packages with an :after clause. @jwiegley perhaps this line of thought might trigger a light-bulb moment for you?

[aspiers]

Here is a draft of the kind of macro I envision:

(defmacro with-packages (when &rest args)
  "Allow declaration of `use-package' config which is only
triggered when a required set of packages are loaded.  The
required set is defined by the `when' argument, whose value is
exactly the same format as with the `:after' argument to
`use-package'.

The `args' are passed straight to `use-package' for use as normal.

Example usage:

  (with-packages (org counsel)
    :bind (:keymap org-mode \"C-c C-j\" . counsel-org-goto))"
  (let ((pseudo-pkg-name
         (format "*with-packages/%s"
                 (replace-regexp-in-string
                  "(\\(.+\\))" "\\1"
                  (s-replace-all '((" " . "-") (":any" . "any") (":all" . "all"))
                                 (prin1-to-string when))))))
    `(use-package ,pseudo-pkg-name
       :no-require t
       :ensure nil
       :after when
       ,@args)))

There are some issues with this - for example it can declare packages with names like *with-packages/any-(all-a-b)-(all-c-d) which I'm not sure is ideal. Also I guess it should avoid a dependency on s.el, but that's easily fixed.

[conao3]

use-package is itself an advanced DSL, so build more DSL on top of it I disagree with that. If you want to do it, As @jwiegley says, you'll want to make another package. I guess.

[aspiers]

I just realised that this is also needed to get nice indentation:

(put 'with-packages 'lisp-indent-function 'defun)

[aspiers]

OK, my with-packages macro is approaching something vaguely functional. If anyone wants to try it, it's here:

https://github.com/aspiers/emacs/blob/master/.emacs.d/lib/with-packages.el

[jwiegley]

Very cool @aspiers. Btw, here is how I would do that:

   (use-package org-counsel
     :no-require t
     :after (:and org counsel)
     :bind (:map org-mode-map
                 (("C-c C-j" . counsel-org-goto))))