straight.el
straight.el copied to clipboard
radian-software
Options for async repo processing, deferring git actions
I get the package doesn't even have a proper release yet, so for the future: It would be nice to have an option for the repo processing (pulling, normalizing, etc.) to be done asynchronously. As it is currently, with a lot of packages you sorta just have to sit and wait it out, which is mildly tedious.
Along the same vein, it would be cool if one could "defer" the pop up asking what to do, that is, go through and pull and stuff, but don't ask, simply notify that an action can be taken. Then the user could call a function or something that prompts for what to do with the repos where an action is needed.
Good idea. It would be rather complicated to do this for the initial clone (as it would require the user to put all their package declarations in one place), but the bulk repository management operations could certainly have this done.
To your second suggestion, I agree that this would be interesting. It would however require a rethinking of how repository management works. The way it currently goes is in two layers. The first layer is a mapping over all the repositories that keeps track of skipped and canceled repositories; this layer calls some function to perform an action on any given repository. The second layer is a loop that repeatedly checks a series of conditions. If any of the conditions are not satisfied, some action is taken (possibly via a popup), and the loop restarts from the beginning. The reason it's done this way is because later validation steps (like doing a checkout) might violate earlier checks (like making sure the working directory is clean). So it's not that each repository has a single action run on it; it's actually much more sophisticated than that.
If you want to take a look at the code, the first layer is straight--map-repos-interactively and the second layer is the straight--vc-git-* functions.
Let me be clear though that I think your ideas are good and I'd like to see them implemented. It's just that the conceptual complexity means that without someone providing a pull request, I probably won't be able to get to them for a while :)
What are these "actions to be taken"?
If we want installation to always be declarative and for init files to always be the source of truth, these actions should get variables inside use-package and straight-use-package and simply be declared up front. Then everything should be able to be moved to non-blocking, yes?
What are these "actions to be taken"?
- Updating packages.
- Clearing local changes and resetting back to the canonical state.
- Pushing local changes so they can be made canonical.
These actions modify state which is orthogonal to what is declared in the init-file. Note, straight.el does not enforce that you use it in a fully declarative way, and indeed such a thing would be limiting (as it would obviously prevent any local development of packages). Instead, it is declarative by default, but also allows you to make arbitrary local changes, and then provides a command for reviewing/reverting all such changes.
As inspiration for what fast package updates can look like, vim-plug does it well in my opinion. A buffer like that with dedicated mode and hotkeys might be better for async and after-the-fact decision-making than the minibuffer.
@sooheon I thought about this for a while, and I think this is indeed the best option. It would be possible to start off all package updates in parallel, and display the statuses, one line per package. Then there could be hotkeys for showing the popup for the package for the current line, for mapping over multiple lines, or for showing a single popup for all lines with a certain status and then applying the same action to all of them. This would eliminate more or less all the UX problems of the popup workflow itself, although there are still things like #54 to consider.
FWIW I think this is a neat idea that pretty much addresses both of my requests.
Would this mean a dependency on async.el though? And would that be an issue (for bootstrap)?
I don't know to what extent Emacs is capable of running things asynchronously. I know you can do that to some extent by setting timers and such, and processes are asynchronous by default.
If async.el was needed, then the current bootstrap can in fact accommodate that (it's magic), but it would be a little undesirable because then the user would only be able to override the recipe for async.el using straight-recipe-overrides, a violation of Principle of Least Surprise.
Another thing I just realized and hadn't considered, the async library is gpl2+, so to use it you'd be legally required to 'relicense' straight.el under gpl2 or gpl3.
straight already uses eg cl-lib which is GPL3+, so if using GPLd libraries requires GPL license shouldn't straight be relicensed regardless of using async?
Yeah I thought of that, but then because it says that they're (subr-x too) part of emacs, maybe it could be thought that emacs is the 'platform' for straight, so straight isn't subject to the GPL in that regard even though it's using them. I don't know if that makes any sense.. but yeah I think technically it's still "using a library". But at what point do you draw the line? Does that mean that any emacs package that uses a built-in GPL library has to be GPL too? I think probably yes, but I'm not sure.
Update: read @SkySkimmer's comment here: https://github.com/raxod502/el-patch/issues/12#issuecomment-337846081. It seems like it's not necessary to relicense to the GPL.
Edit: Although I should make clear that that case probably doesn't apply to async.el (only cl-lib and subr-x), since async.el doesn't come with the "interpreter" [emacs]. So to use async.el, I think you would still have to relicense to the GPL. @SkySkimmer would you say that's correct?
In my opinion, dynamic linking is not covered by the GPL. As the FSF's claim to the contrary has not been tested in court, it is my opinion against theirs.
Therefore, I believe straight.el can use any dependencies it needs—including async.el—as long as they are not distributed along with straight.el.
Therefore, I believe
straight.elcan use any dependencies it needs—includingasync.el—as long as they are not distributed along withstraight.el.
Just to let you know, I don't think this is the case: https://www.gnu.org/licenses/gpl-faq.en.html#GPLStaticVsDynamic
As in, neither static nor dynamic linking is covered by the GPL. But, for example, that wouldn't mean that static linking wouldn't involve creating a derivative work. I think, regarding the GPL, static vs dynamic linking is like 'an implementation detail', as both involve creating a derivative work, due to use of the library.
[...] as long as they are not distributed along with
straight.el.
Perhaps you are confusing the GPL with the LGPL? Because with the latter as I understand it you can dynamically link against a LGPL library without needing your application to be LGPL: https://www.gnu.org/licenses/gpl-faq.en.html#LGPLStaticVsDynamic. (In fact, as it says there, you can do link to it statically too without requiring your application to be LGPL, but you must provide your application in at least an 'object' format.)
regarding the GPL, static vs dynamic linking is like 'an implementation detail', as both involve creating a derivative work, due to use of the library
… in the opinion of the FSF, but not in my opinion. As I said, their claim that dynamic linking constitutes a "derivative work" has not been tested in court.
What you have said is entirely correct according to the FAQ, indeed; it is just that I think the FAQ is partisan conjecture with no legal significance.
Returning to the pragmatic side of things—in the discussion about licenses of Emacs packages on emacs-devel, I did not recall seeing anyone saying that Emacs packages must be GPL'd just because they are "dynamically linked" with GPL'd Emacs libraries.
Yeah you're right, I thought you didn't know about that FAQ answer and so wanted to point it out. And I thought that since the FSF created the GPL it would be a proper reference to use, but you're right it hasn't been tested in court and it's debated whether dynamic linking really constitutes creating a derivative work. I wouldn't call that discussion proof that emacs packages don't have to be GPL'd when using GPL elisp libraries, it seems to just mainly turn into a discussion on how to address the situation with the packages that are unlicensed?
@vyp i understood the point isn't so much that there's proof they don't have to, but that there's no proof they have to, hence not having been tested in court.
Looks like someone else had a similar idea for the interface of a repository management too: https://github.com/thblt/borg-queen
Talking as just a user, my dream scenario would be for straight to quietly fetch repositories in the background (with a configurable interval), and pop up a notification when packages can be upgraded. And the dedicated buffer to manage the upgrade.
Everything else is auto-upgrading these days, so it seems a bit old-fashioned to have to check for updates manually (nevermind the time it takes to pull a bunch of repos).
@xendk I'm not opposed to including such behavior—as an optional extension—if someone were to implement it. How to actually do that requires some thought, however, since the local repositories can have arbitrary changes (which is why straight-fetch-package runs straight-normalize-package first).
@raxod502
Shouldn't doing git fetch be quite safe and not need normalization? It should take the bulk out of the process of updating. Then a straight-update-packages could check if there's packages where the origin HEAD has moved, and work from that.
@xendk Yes you are most likely right. (Although use git fetch --all, most likely.) I think that is a great idea. The complexities are probably mostly to be found in the user interface.
@raxod502
And now I realize that there already is straight-fetch-all and straight-merge-all functions, which does improve the situation over straight-pull-all (at last you can do something else while fetching).
Refactoring straight-pull-all (and straight-pull-package of course) to use straight-fetch-* and straight-merge-* makes sense to me (they should be functionally equivalent, right?), and if straight-pull-all ran straight-fetch-all before straight-merge-all, it too would become more bearable. Could remove the pull operations from the VC backend interface to simplify things.
Then there's the challenge to make a version of straight-pull-all that can run in the background (magit has inspiration there).
After that, one might consider an straight-update-all that works like straight-merge-all but just does fast-forwards, only asking for directions in the hairier cases. Not quite the interactive buffer previously mentioned, but most likely what people need 95% of the time.
I'll take a stab at something of the above when I have some free time, but that shouldn't stop anyone else.
what´s the status of this? straight froze Emacs for several minutes due to emacs-emojify being 27.81 MiB in size and my Internet connection sucking a lot and because there is no much feedback from straight you don´t know if something went wrong or is just taking time
It would be nice. As a workaround I've taken to building my Emacs separately in Nix, with the packages defined there. Then I just use straight.el to configure them. In the same way, you can have one Emacs process build the package while you use another to work.
I agree that a well-placed dotimes-with-progress-reporter could help make user feedback better during downloads.
What about using an async shell command to wrap the long running processes? If I run
(async-shell-command "emacs -batch -l ~/.emacs.d/init.el -f straight-pull-all")
It seems to mostly work. However, straight--popup doesn't seem to be designed to run in batch mode so any repos that need decisions made are just skipped I think.
Is there currently a way to do all the pulling and merging in a separate process? Is implementing a batch mode in straight--popup the only blocker or are there other issues?
@mnewt I don't think using a subprocess is a good idea, because then you have to deal with a lot of additional complexity. The thing that takes time is running the commands, and that can be done asynchronously within the current Emacs session. I think the "best" solution would be to make it possible to run the repository management commands asynchronously. For example, you could create a macro that would transform every subprocess call into a make-process invocation with a callback -- much like the async / await implementation in JavaScript.