Alternative prompts for any function

[Hugo-Heagren]

Fix #528

I'm really proud of this.

Motivation

From the above issue:

The basic idea is that I could call some embark action during any call to read-file-name, be presented with a list of possible alternatives (some of which I might have defined myself), like with embark-become, select one, type something, hit enter, have the function I chose return something from that input, and have that return value returned to whatever function initially called read-file-name.

That's the abstract version. Here's a motivating use-case. I'm a student, and I often need to send academic papers to other people at my university. As a good amateur archivist, I name all my pdfs with the key of the relevant bib entry in my bib file. This makes it very easy for ebib and citar to play nicely with my pdf library, which is great. But if I want to attach a file to an email, I have to open ebib, find the record, copy its key, switch back to the email, attach a file (which calls read-file-name), then navigate to my pdf dir and yank the key into the minibuffer. It would be very easy to hack together a function from the ebib or citar codebases which reads a paper name and returns the filepath to it, but I wouldn't be able to use this directly in my email composition -- thus the idea of integrating with embark using functionality similar to become.

Functionality

What I've written is actually more generic -- a general way to use alternative prompts in functions which prompt for input in the minibuffer. Think of it like embark-become, but for the current prompt, not the whole command.

The user experience is heavily inspired by embark-become:

• When in a minibuffer prompt, run embark-alternative-input (from an embark prompt or otherwise)
• If any of the functions in the current backtrace appear in embark-alternative-input-alist, then present the keymap of the first one which does. This can be used to select an function. Functions can also be run from M-x. (see shortcomings below)
• This function should return a string, which is then inserted into the minibuffer.

This string can have text properties which determine whether we then run exit-minibuffer. This is because the idea of this feature is for users to write their own functions which are useful for their own workflows, so I wanted it to be possible to write powerful DWIM commands (hence the exit/don't exit customisability), while also leaving it fairly easy for users who don't know much elisp to write something (hence using strings rather than a more complex data structure as return values). The default is to not exit, and there is a prefix argument on the command which changes this.

Working example

This should run on any recent Emacs, and demonstrates using bookmarks as input, as well as useful DWIM functionality with the text properties.

(defun my/filename-from-bookmark ()
  (interactive)
  (let ((file
	 (bookmark-get-filename
	  (bookmark-completing-read "Filename for bookmark"
				    bookmark-current-bookmark))))
    ;; Accept if a file, allow edit if a directory
    (propertize file 'embark-replace-input-accept (not (file-directory-p file)))))
	  
(embark-define-keymap my/embark-alternative-input-read-file-name-keymap
  "Keymap for `read-file-name' alternative inputs.
Functions must return a string (in particular,they should return
\"\", rather than nil, as a 'null' return value). See also
`embark-alternative-input'."
  :parent nil
  ("m" '("Filename from bookmark" . my/filename-from-bookmark)))
	
(setq
 embark-alternative-input-alist
 '((read-file-name . my/embark-alternative-input-read-file-name-keymap)))

Shortcomings

Some things I think would be cool, but I can't make work:

I've bound embark-alternative-input to C-r in embark-general-map. Is this an alright key?

Target cycling would be great. Say I have different alternative input keymaps set up for org-msg-attach and read-file-name. org-msg-attach calls read-file-name, so when I call embark-alternative-input, read-file-name will be the first function it recognises as relevant in the backtrace (the "target"). This is fine, but there is currently no way to say "no, I want the next relevant function back in the trace" (which in this case would be org-msg-attach). I've had a brief look at the code for embark-cycle and found it a bit impenetrable. Would be happy to have a go at implementing this if someone gave me some pointers though.

I think error and exit handling (signalling "Cancelled" at the right time etc.) is correct, but it would be good if someone else could check this especially.

[minad]

I have difficulties to understand what you are trying to achieve here, but it reminds me of consult-dir and consult-history. If you run consult-dir it will replace the current input with some other string from another source. But in both cases the functionality is not really generic, such that it would fit into Embark. What is different here? Or did I misunderstood your idea?

Generally I would prefer if we take it slowly with adding new functionality to Embark. Personally I am also not fond of embark-become for example which I use almost never. It has been critized before e.g. by @karthink that Embark is growing in complexity. We made steps towards simplification and consolidation, for example the removal of the Embark completion UI.

One thing I consider as a red flag here is that you add a command and the embark-alternative-input-alist configuration variable. But the variable is empty! Compare that to the embark-act or embark-become variables which come with a rich configuration. Therefore it seems that the added functionality is not generally useful. As an arbitrary threshold, I think there should be at least three such alternative input keymaps with at least 10 commands in total. If we have that as a data basis it will also be easier to decide if this feature is needed or if it can be integrated with the existing functionality in another way.

I hope my critical feedback doesn't come over the wrong way. It is Omar's project anyway :)

[Hugo-Heagren]

Good feedback -- thanks.

I have difficulties to understand what you are trying to achieve here

I think I should have said more about this earlier.

At a high level, I want to be able to reduce a common pattern where:

• a command I have invoked requires input in a certain format
• the easiest or preferred way of working out what to input is inconvenient in that format

To solve this, this PR allows you to present the same or equivalent information, in a more easily accessible way, just for that prompt.

Attaching a research pdf to an email is a good example. To attach any file, I run org-msg-attach (I use org-msg, I'm sure similar things apply to other emacs mail packages). This requires a filepath input. I name my pdfs in a way that makes it very difficult to immediately see which file is the one I want. Instead, I can use a prompter (built with citar) which presents the same information, in a more easily-accessible way. The prompt on the screen now corresponds with the way that I think about what I'm doing -- choose a paper from my library -- not the way that the attaching function understands it (choosing a file from a tree). Implementing this in a generic way in embark means that.

Another good example might be emacs patches. I made my first contribution to org-mode recently, and had to learn a new workflow with saving a patche to a file, then writing an email to which I attached this file manually. It wasn't hard work, but it was an extra step in the process and it slowed me down. With this PR, I could write a function which prompts for a recent directory or project, then uses magit's api to prompt for a commit range, save the patch to a temp dir, and return the resulting filepath.

It's worth considering how else we might solve this problem, and why I don't like that approach. I think the standard way in emacs to solve either problem is to write new commands, which would be invoked instead of org-msg-attach: my/attach-research-paper and my/make-and-attach-patch. This works (and will include about the same amount of initial work to write the code), and for some people (@minad?) this is the best way forward.

There are two things I don't like about this. First, it adds cognitive effort at a place in the system I don't want it. Now when attaching to an email, I have to choose which function to run, and then choose again which thing to attach (in the prompt the function presents). Personally when attaching anything to an email, I want to be able to just hit one key sequence, the same one every time, and worry about finer points as I start finding the file. I want the cognitive effort to all be in one place -- the file finding interface. I think embark-become exists for a similar reason -- some people would rather run a command and realise later that they needed a different one, than run the right command to begin with. I recognise that this isn't everyone's way, but one of the great things about emacs is how it can accomodate many different workflows (see point under the next quote).

Second, this limits what I can do with the research-paper prompt, to only attaching to emails. That is what the command my/attach-research-paper will do: it will prompt me for a paper and attach it to the current email. The PR separates these from each other. If I later user a different command which calls read-file-name, I can use the research paper prompter for that too (e.g. if I start using a different emacs email client, or an emacs reddit/stackexchange client, where I might have to choose files to upload). Very roughly: this PR lets me define new, more desirable ways of selecting a file (or doing anything else which requires a prompt) without having to edit the basic commands (which are often quite complex, see read-file-name-default).

One thing I consider as a red flag here is that you add a command and the embark-alternative-input-alist configuration variable. But the variable is empty!

I did worry a bit about this myself, but I'll explain why I left it empty in the end (note also I'm very happy for this PR to be a jumping-off point, and change a lot before being accepted).

Both of the examples I gave above are pretty pretty specific to me and my setup/tools. Even if you have a problem like mine with the research papers, you might use ebib more than citar, and thus prefer to use its resource selection functions. And all of the packages I've mentoned are external to core emacs and embark -- none of these functions would be useful to anyone who hadn't installed them. I left the variable empty because I think the general functionality will be most useful to users who want to write their own functions. I believe it's worth providing the structure for them to work within -- but I can't imagine many applications which would definitely be useful for many users and use only functions from emacs core and embark.

Therefore it seems that the added functionality is not generally useful.

So, the short answer to this is that the added functionality is the ability for users to define and use alternative prompts -- not any particular alternatives. I think this is useful (I for one would certainly use it).

consult-dir and consult-history

That said, I was heavily inspired by consult-dir. I just wanted something a bit more flexible, and I think this might be useful for people who don't use consult. Perhaps the bookmark picker above, and some corresponding to the other sources in consult-dir would be useful for lots of people, and use only available resources.

Hope that's helpful!

[minad]

Thanks for the detailed responseand the clarifications. I understand that there is a general pattern and that it may be worth to pursue this in the general setting of Embark. consult-dir and consult-history are both examples of useful alternative inputs, as is Citar or some other file source.

As I see it my criticism still stands. Let me write down a more detailed list:

1. I really miss here a set of generally useful preconfigured alternative input commands, which demonstrate the usefulness beyond a handful of use cases. I understand that many use cases lie outside of Embark. But as I understand Embark is not only meant as an entry point for external functionality. We have to think about the 99% of the users not only about some special needs. You should consider that every addition has some weight which makes this package more difficult to get in.
2. Embark aims to make completion commands reusable. This aspect is also missing here since it seems to me that for every alternative input you have to write a separate special alternative input provider. In my opinion the reuse aspect is what makes Embark truly great. Both embark-act and embark-become just reuse existing commands. Alternative input does not.
3. I wonder if there is even a need to make this part of Embark? You reuse the Embark indicators but that's it. Why not package this up separately like consult-dir? It is a good thing to have a modular ecosystem. If something doesn't fit perfectly it is better developped separately.
4. You mention that you want a single entry point for your commands. You could still bind such a command in the appropriate minibuffer maps without relying on anything from Embark.
5. The backtrace-based approach seems brittle. Also it is unlike current techniques used in Embark.
6. More conventional approaches may also work for most users, e.g., writing separate commands or actions.

All these points lead me to believe that this functionality is not a great fit here. The main argument which I think is in favor of this is that there is a useful pattern - as demonstrated by consult-dir. Question remains if it is general enough, which has not been demonstrated.

Did you consider developing this separately of Embark as a new package?

[minad]

To address the completion command reuse aspect - I could imagine a facility where you run an alternative completion command and then instead of invoking the continuation of completing-read you magically steal the input and redirect it to the original command. This would be very Embarky but maybe also a bit convoluted. Furthermore one would also have to check if there are many (built-in) commands which would make sense as alternative input providers.

[minad]

Thinking about this more - one thing I would explore more is why "conventional" approaches don't work for you or for this special use case. Is there not a way to achieve a satisfactory workflow with separate commands or with some clever use of prefix maps? The reason why I consider this important is that you are not focusing much on the command reuse aspect which would justifies a general mechanism in the style of Embark. Also I find it a good idea to stick to builtins if possible, this means pushing "conventional approaches" as far as possible.

Anyway the general pattern of alternative input is worth to be explored more. I always felt that consult-dir and consult-history are not the end of the story here.

[karthink]

The core idea here is very good with a lot of potential. This PR attempts a general solution to the problem that sparked the creation of consult-dir, so it got me thinking.

But as I understand Embark is not only meant as an entry point for external functionality

However, I agree with @minad that this doesn't fit Embark's design or common use-cases.

This is in part because the user is expected to write or find a collection of (completing-read) commands that return strings for the purpose of inserting into the minibuffer. Command reuse can proceed only after these are available.

You can't reuse any existing command for this purpose unless they return a string with no side-effects (very rare for interactive commands).

Second, the behavior of this PR is very different from Embark's standard pattern: embark-act and embark-become replace running commands with other commands at the same level of the command loop. IIUC, this is true even when embark is set to not quit after an action. In comparison, this code runs at two recursive-edit levels. For this reason, it can't reuse embark-act or embark-become, or make direct use of the rich options defined in the action and become keymaps. (embark-file-map, embark-become-file+buffer-map, etc.)

Third, Embark tends to get harder to understand or explain to users with each new feature that does something different. Of course, this is Omar's call. But it's ironic for a package whose raison d'être is composability :)

Anyway the general pattern of alternative input is worth to be explored more. I always felt that consult-dir and consult-history are not the end of the story here.

As a generalization of consult-dir and consult-history, I think this PR is on the right track.

If I understand correctly, this code is a combination of two or three independent features. I played around a bit with the ideas to see if I could find an implementation that

• is Embark-independent,
• doesn't mandate extra configuration like embark-alternative-input-alist,
• avoids looking at backtraces,
• and works with all existing Emacs commands (irrespective of return types) like Embark does.

Feature 1: "Steal input and redirect it to the original command"

I could imagine a facility where you run an alternative completion command and then instead of invoking the continuation of completing-read you magically steal the input and redirect it to the original command.

(defun minibuffer-replace-input ()
  (interactive)
  (when (and (minibufferp) (> (minibuffer-depth) 1))
    (let* ((replacement (minibuffer-contents)))
      (unwind-protect (minibuffer-quit-recursive-edit)
        (run-at-time 0 nil
                     (lambda (rep)
                       (delete-minibuffer-contents)
                       (insert rep))
                     replacement)))))

(define-key minibuffer-local-map (kbd "C-x C-i") 'minibuffer-replace-input)

(Be sure to enable lexical-binding.)

With this, you can do something like the following:

1. Start a minibuffer session. This is the "original command".
2. Run an (unrelated) second "child" minibuffer command and choose a completion.
3. Press C-x TAB or C-x C-i to replace the input of the original command with this completion.

You can continue to propagate the chosen input up the recursive-minibuffer stack.

This should work with all Emacs minibuffer-based commands, and the commands don't have to return strings.

Personally when attaching anything to an email, I want to be able to just hit one key sequence, the same one every time, and worry about finer points as I start finding the file. I want the cognitive effort to all be in one place -- the file finding interface.

This feature solves the "cognitive effort in one place" problem, modulo assigning an automatic "child" minibuffer command to each "original command" and binding it to a key.

Second, this limits what I can do with the research-paper prompt, to only attaching to emails. That is what the command my/attach-research-paper will do: it will prompt me for a paper and attach it to the current email. The PR separates these from each other. If I later user a different command which calls read-file-name, I can use the research paper prompter for that too.

It also solves this problem, since you're free to call the "child" command with any other command.

Feature 2: (Not implemented) Transform candidates to an appropriate format

Unfortunately, minibuffer-replace-input alone doesn't solve the general practical problem that @Hugo-Heagren is addressing.

But if I want to attach a file to an email, I have to open ebib, find the record, copy its key, switch back to the email, attach a file (which calls read-file-name), then navigate to my pdf dir and yank the key into the minibuffer.

The completions offered by, say, bookmark-jump, ebib(?) or citar-open-library-file are not file paths but some other kind of reference to each file, like a bookmark name or bib entry metadata. So inserting it verbatim into the original minibuffer prompt (mml-attach-file etc) in the following way does not work well:

1. Call mml-attach-file (or org-msg-attach etc)
2. Call the relevant ebib command and select your key.
3. Press C-x TAB

This inserts the key and not the corresponding PDF file into the attachment prompt. Although it does save you the copy-yank business in the above quote.

What's missing in general is a "transformer" or "input provider" function like @Hugo-Heagren's my/filename-from-bookmark above, or like consult-dir--pick from consult-dir.

In some cases this is not needed. For example, for the attachment problem described above, I could do the following:

1. Compose an email, call mml-attach-file or equivalent.

2. At the file selection prompt, call citar-open.

3. This gives me a pretty listing of my bibliography. Select a resource:

4. This shows the resources for this entry, which includes the file -- the saving grace of this approach! Tab-complete and press C-x TAB.

5. The file's chosen and ready to attach.

(I don't know why the Attachment prompt changed from "Attach file: " to "Find File: ", but the attachment works as expected.)

No Embark or special configuration was needed here, but this is not a general solution. It wouldn't work here:

...a new workflow with saving a patche to a file, then writing an email to which I attached this file manually. It wasn't hard work, but it was an extra step in the process and it slowed me down. With this PR, I could write a function which prompts for a recent directory or project, then uses magit's api to prompt for a commit range, save the patch to a temp dir, and return the resulting filepath.

The solution in the PR is to write custom transformer functions, but that's not very "Embarky", as @minad put it. Writing bespoke "transformers" or "input providers" has this problem:

Embark aims to make completion commands reusable. This aspect is also missing here since it seems to me that for every alternative input you have to write a separate special alternative input provider.

I get the feeling there's an interesting solution waiting to be found for this. Perhaps if elisp had stricter typing this would be a more tractable problem.

Feature 3: (Not implemented) Easier access to "input provider" commands - backtrace checking

If quick access to the "input providers" is desired, I think it makes more sense to offer it through the completion category. If used as part of or with Embark, perhaps find a way to reuse the become and action keymaps (activated through the completion category) instead of looking at the backtrace? As a standalone package, it might be better to simply associate "input providers" with individual commands (org-msg-attach) in an alist instead of looking for lower level commands (read-file-name) in the backtrace. I'm not very knowledgable about this part, just thinking out loud here.

[Hugo-Heagren]

Thanks for the feedback @karthink!

Embark aims to make completion commands reusable. This aspect is also missing here since it seems to me that for every alternative input you have to write a separate special alternative input provider.

I get the feeling there's an interesting solution waiting to be found for this. Perhaps if elisp had stricter typing this would be a more tractable problem.

I am actually fairly convinced this is impossible, because:

• the commands which invoke these completions in the first place are arbitrarily complex
• it is beyond elisp to predict which parts of that arbitrary complexity are relevant.

For example, a general solution for the citar case would have to:

1. call citar-insert-citation
2. somehow arrange for the input to that prompt to be returned to the body of citar-insert-citation (so that any relevant formatting/wrangling can go on)...
3. ... but do this in such a way that only formatting happens, and not anything that citar-insert-citation does
4. somehow extract the result at this point (from partway through citar-insert-citation).

The major sticking point is how we decide what is relevant (2) and what it is not (3).

@karthink can I ask why you don't like the idea of looking at backtraces in general (apart from the fact that you think completion categories make more sense -- you seem not to like matching on backtrace elements?). Similarly @minad, what do you mean when you say backtraces are brittle?

[minad]

@karthink I like the idea of your minibuffer-replace-input command, which propagates input upwards. I think this could fit Embark with a few changes.

1. In the recursive minibuffer minibuffer-replace-input should obtain the current candidate instead via the usual Embark candidate function.
2. The candidate can be transformed by the usual candidate transformers to a canonical form.
3. Embark can check if the resulting category fits the category of the original command (some kind of strict mode). But since we are just copying strings, one could also skip the check.

Overall this seems like a very light addition without additional configuration. However I would still like to see a longer list of command combinations, ideally including a few builtins:

• Org-msg attach -> Citar
• ...?

Also with this approach @Hugo-Heagren will probably miss convince since one has to trigger the nested command via a top-level binding and then invoke the extra command to replace input. The alternative here would be to introduce extra keymaps. The commands specified there will be wrapped such that their continuation after the minibuffer is redirected to the original command.

[minad]

Hmm, the way I described the workflow in my previous comment is just the reverse workflow of embark-act. Instead of Citar->act->attach you do attach->Citar->replace-input. Given that I don't think such an addition is justified. Every addition carries weight and complexity and the benefits are rather small.

[oantolin]

Sorry for taking so long to get to this, @Hugo-Heagren. I quite like this idea as an abstract idea but don't think it is a good fit for Embark, basically for reasons already mentioned above by @minad and @kathink: the user would need to write their own prompt/transform string commands, and I don't really get the feeling that many generally useful ones are possible. I encourage you to instead publish this or something like it as a separate package, specially if you can come up with some more generally useful examples of its usage.