elpaca
elpaca copied to clipboard
Published
20 hours ago •
progfolio
progfolio
[Feature]: elpaca-process-poll unnecessarily creates a emacs subprocess
Feature Description
elpaca-process-poll currently runs a emacs subprocess which runs the target process.
A subprocess is already sufficiently isolated so this is not needed.
Confirmation
- [x] The feature I'm proposing does not already exist in Elpaca
You're correct. The only place this is currently used is in elpaca--run-build-commands.
Eventually I'd like to completely remove the :pre-build and :post-build recipe keywords, replacing them with tools which make it easier to manipulate the :build keyword. That should be conceptually simpler.
From using the build commands, I have found the following problems:
(1) It always says it's successful. I had to modify elpaca-process-poll to make this otherwise; Hence this issue
(2) It's not possible to transfer lisp state between lines in the :build. This means that you want to pass some lisp evaled state into your build step or to a later lisp step, you need to have a single lisp form and call commands from there (I invoked them throngh a elpaca function since invoking processes in emacs is a pain)
What do you plan to replace build with?
From using the build commands, I have found the following problems:
(1) It always says it's successful. I had to modify
elpaca-process-pollto make this otherwise; Hence this issue
Right. The command was original using pattern based error detection as a workaround.
elpaca-process-poll--filter still has vestigial PATTERN and ERROR parameters, but the call is baked into elpaca-process-poll without making use of them.
(2) It's not possible to transfer lisp state between lines in the
:build. This means that you want to pass some lisp evaled state into your build step or to a later lisp step, you need to have a single lisp form and call commands from there (I invoked them throngh a elpaca function since invoking processes in emacs is a pain)
Could you show an example of the state you'd like to pass and how you're going about it now?
What do you plan to replace build with?
I don't plan on replacing :build. Just removing :pre-build and :post-build, as they are special cases of :build.
I think you should keep pre-build and post-build. They are useful for building a dynamic module or installing some external dependency into the build directory.
Rethinking the passing state, what I currently have is actually the cleanest. I now to something simpler but I used to have:
'(package
:pre-build
(let ((cond (computed)))
(when cond
(do-lisp))
(elpaca-process-poll "run-cmake")
(when cond
(do-lisp))))
which I think is a better interface. Perhaps you can deprecate commands and just have the user write a lisp form; they can call a function to run commands.
Oh perhaps we shouldn't use a subprocess for pre-build at all. It will be very nice if we can do:
'(package
:pre-build
(setq byte-comp-config 'stuff)
(setenv "envvar_used_by_package_during_compile" "stuff"))
I think you should keep
pre-buildandpost-build. They are useful for building a dynamic module or installing some external dependency into the build directory.
I was thinking more along the lines of more convenient ways to alter the :build step list. e.g.
;;Remove "removed" step from the defaults and add a pre-build, post-build step.
(elpaca (example :build (pre-build (- defaults removed) post-build)))
which I think is a better interface. Perhaps you can deprecate commands and just have the user write a lisp form; they can call a function to run commands.
It wouldn't be difficult to write a DSL which allows for forms or functions. It's just a matter of thinking through a design that is performant, flexible, does not require much configuration for common cases, and reads well in a declaration. There's also the issue of commands which need to alter the build step list. e.g. rebuilding a package, fetching, etc. I need to experiment more to see what works well in practice.
Oh perhaps we shouldn't use a subprocess for
pre-buildat all.
That could be possible via the DSL. It can be done now by creating a synchronous build step. e.g.
Test Case
(elpaca-test
:init (push (lambda (e)
(elpaca--signal e (format "Hello...%s" (elpaca<-id e)))
(sit-for 2)
(elpaca--continue-build e))
elpaca-build-steps)
(elpaca (doct :wait t))
(princ (elpaca-log "doct |")))
Host Env
| elpaca | 722b36c HEAD -> master, origin/master, origin/HEAD |
| installer | 0.9 |
| emacs | GNU Emacs 31.0.50 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.43, cairo version 1.18.2) of 2025-01-26 |
| git | git version 2.48.1 |
Output
INFO Scraping 16 files for loaddefs...
INFO Scraping 16 files for loaddefs...done
GEN ../elpaca-autoloads.el
Cloning into '/tmp/elpaca.kEYhx8/elpaca/repos/elpaca'...
Your branch is up to date with 'origin/master'.
Checking /tmp/elpaca.kEYhx8/elpaca/repos/elpaca...
Compiling /tmp/elpaca.kEYhx8/elpaca/repos/elpaca/elpaca-info.el...
Compiling /tmp/elpaca.kEYhx8/elpaca/repos/elpaca/elpaca-log.el...
Compiling /tmp/elpaca.kEYhx8/elpaca/repos/elpaca/elpaca-manager.el...
Compiling /tmp/elpaca.kEYhx8/elpaca/repos/elpaca/elpaca-menu-elpa.el...
Compiling /tmp/elpaca.kEYhx8/elpaca/repos/elpaca/elpaca-menu-melpa.el...
Compiling /tmp/elpaca.kEYhx8/elpaca/repos/elpaca/elpaca-menu-org.el...
Compiling /tmp/elpaca.kEYhx8/elpaca/repos/elpaca/elpaca-process.el...
Compiling /tmp/elpaca.kEYhx8/elpaca/repos/elpaca/elpaca-test.el...
Compiling /tmp/elpaca.kEYhx8/elpaca/repos/elpaca/elpaca-ui.el...
Compiling /tmp/elpaca.kEYhx8/elpaca/repos/elpaca/elpaca.el...
Checking /tmp/elpaca.kEYhx8/elpaca/repos/elpaca/doc...
Compiling /tmp/elpaca.kEYhx8/elpaca/repos/elpaca/doc/early-init.el...
Compiling /tmp/elpaca.kEYhx8/elpaca/repos/elpaca/doc/init.el...
Compiling /tmp/elpaca.kEYhx8/elpaca/repos/elpaca/doc/installer.el...
Checking /tmp/elpaca.kEYhx8/elpaca/repos/elpaca/extensions...
Compiling /tmp/elpaca.kEYhx8/elpaca/repos/elpaca/extensions/elpaca-use-package.el...
Checking /tmp/elpaca.kEYhx8/elpaca/repos/elpaca/images...
Checking /tmp/elpaca.kEYhx8/elpaca/repos/elpaca/test...
Compiling /tmp/elpaca.kEYhx8/elpaca/repos/elpaca/test/elpaca-test.el...
Compiling /tmp/elpaca.kEYhx8/elpaca/repos/elpaca/test/elpaca-tests.el...
Done (Total of 12 files compiled, 4 skipped in 4 directories)
Downloading MELPA recipes...
Downloading MELPA recipes...100%
doct queued Hello...doct 02.502139
doct cloning Cloning 04.505245
doct cloning $git clone --filter=tree:0 https://github.com/progfolio/doct.git /tmp/elpaca.kEYhx8/elpaca/repos/doct/ 04.505740
doct cloning Cloning into '/tmp/elpaca.kEYhx8/elpaca/repos/doct'... 04.507852
doct cloning remote: Enumerating objects: 300, done. 04.728247
doct cloning remote: Counting objects: 100% (28/28), done. 04.732462
doct cloning remote: Compressing objects: 100% (28/28), done. 04.732837
doct cloning remote: Total 300 (delta 0), reused 27 (delta 0), pack-reused 272 (from 1) 04.773003
doct cloning Receiving objects: 100% (300/300), 60.09 KiB | 2.31 MiB/s, done. 04.773533
doct cloning remote: Enumerating objects: 4, done. 05.029852
doct cloning remote: Counting objects: 100% (3/3), done. 05.030822
doct cloning remote: Compressing objects: 100% (3/3), done. 05.038829
doct cloning remote: Total 4 (delta 0), reused 3 (delta 0), pack-reused 1 (from 1) 05.039189
doct cloning Receiving objects: 100% (4/4), 674 bytes | 674.00 KiB/s, done. 05.039507
doct cloning remote: Enumerating objects: 14, done. 05.205168
doct cloning remote: Counting objects: 100% (8/8), done. 05.210976
doct cloning remote: Compressing objects: 100% (7/7), done. 05.212010
doct cloning remote: Total 14 (delta 0), reused 4 (delta 0), pack-reused 6 (from 1) 05.241233
doct cloning Receiving objects: 100% (14/14), 53.11 KiB | 1.90 MiB/s, done. 05.242035
doct unblocked No external dependencies 05.246903
doct unblocked Checking dependency versions 05.247588
doct linking Linking build files 05.248053
doct linking Build files linked 05.248946
doct autoloads Generating autoloads: /tmp/elpaca.kEYhx8/elpaca/builds/doct 05.249424
doct autoloads $/usr/bin/emacs -Q -L /tmp/elpaca.kEYhx8/elpaca/repos/elpaca/ -l /tmp/elpaca.kEYhx8/elpaca/repos/elpaca/elpaca.el --batch --eval (progn (setq gc-cons-percentage 1.0) (elpaca-generate-autoloads "doct" "/tmp/elpaca.kEYhx8/elpaca/builds/doct")) 05.250118
doct autoloads INFO Scraping 1 files for loaddefs... 05.331578
doct autoloads INFO Scraping 1 files for loaddefs...done 05.336886
doct autoloads GEN doct-autoloads.el 05.337535
doct autoloads Autoloads Generated 05.337920
doct byte-compilation Byte compiling 05.338278
doct byte-compilation $/usr/bin/emacs -q --batch --eval (let ((gc-cons-percentage 1.0) (native-comp-eln-load-path '("/home/n/.emacs.d/eln-cache/" "/usr/lib/emacs/31.0.50/native-lisp/"))) (dolist (dir '("/tmp/elpaca.kEYhx8/elpaca/builds/doct")) (let ((default-directory dir)) (add-to-list 'load-path dir) (normal-top-level-add-subdirs-to-load-path))) (byte-recompile-directory "/tmp/elpaca.kEYhx8/elpaca/builds/doct" 0 'force)) 05.338658
doct byte-compilation Checking /tmp/elpaca.kEYhx8/elpaca/builds/doct... 05.403720
doct byte-compilation Compiling /tmp/elpaca.kEYhx8/elpaca/builds/doct/doct-autoloads.el... 05.411175
doct byte-compilation Compiling /tmp/elpaca.kEYhx8/elpaca/builds/doct/doct.el... 05.412251
doct byte-compilation In doct--first-in: 05.421284
doct byte-compilation doct.el:217:18: Warning: ‘when-let’ is an obsolete macro (as of 31.1); use ‘when-let*’ or ‘and-let*’ instead. 05.422338
doct byte-compilation In doct-get: 05.431407
doct byte-compilation doct.el:301:6: Warning: ‘if-let’ is an obsolete macro (as of 31.1); use ‘if-let*’ instead. 05.432226
doct byte-compilation In doct--target: 05.432622
doct byte-compilation doct.el:385:7: Warning: ‘if-let’ is an obsolete macro (as of 31.1); use ‘if-let*’ instead. 05.433020
doct byte-compilation In doct--additional-options: 05.433407
doct byte-compilation doct.el:583:8: Warning: ‘when-let’ is an obsolete macro (as of 31.1); use ‘when-let*’ or ‘and-let*’ instead. 05.433790
doct byte-compilation In doct--run-hook: 05.434166
doct byte-compilation doct.el:601:4: Warning: ‘when-let’ is an obsolete macro (as of 31.1); use ‘when-let*’ or ‘and-let*’ instead. 05.434539
doct byte-compilation In doct--add-contexts: 05.434919
doct byte-compilation doct.el:683:4: Warning: ‘when-let’ is an obsolete macro (as of 31.1); use ‘when-let*’ or ‘and-let*’ instead. 05.435297
doct byte-compilation doct.el:688:10: Warning: ‘if-let’ is an obsolete macro (as of 31.1); use ‘if-let*’ instead. 05.435684
doct byte-compilation In doct--compose-entry: 05.436075
doct byte-compilation doct.el:734:22: Warning: ‘when-let’ is an obsolete macro (as of 31.1); use ‘when-let*’ or ‘and-let*’ instead. 05.436813
doct byte-compilation In doct--convert: 05.438034
doct byte-compilation doct.el:761:16: Warning: ‘when-let’ is an obsolete macro (as of 31.1); use ‘when-let*’ or ‘and-let*’ instead. 05.438465
doct byte-compilation Done (Total of 1 file compiled, 1 skipped) 05.438881
doct byte-compilation Byte compilation complete 05.439322
doct info Compiling Info files 05.439744
doct info Info source files not found 05.440160
doct activation Activating package 05.440669
doct activation Package build dir added to load-path 05.441116
doct activation Caching autoloads 05.441571
doct activation Autoloads cached 05.442182
doct finished ✓ 2.946 secs 05.442655
Test Case How to run this test?
I think this doesn't help (I didn't test); It sets variable or env var on the main emacs process. I would like to set theses just when compiling that package; I would certaintly not like env var to leak into what I am developing with emacs.
Perhaps we can do "clone" and "fetch deps" on the main process, and do everything else on a seperate one, not isolating them from each other?
I would like to set theses just when compiling that package; I would certaintly not like env var to leak into what I am developing with emacs.
The :vars recipe keyword could be used in that case.
:vars wont work for bytecomp and :pre-build since they are in a subprocess.
I suppose the intention is for it to propagate?
Test Case
(elpaca-test
:ref "722b36c"
:keep t
:init (elpaca (general :wait t :vars ((byte-compile-warnings '(not docstrings)))))
(princ (elpaca-log ".*")))
Host Env
| elpaca | 28454ae HEAD -> master, origin/master, origin/HEAD |
| installer | 0.9 |
| emacs | GNU Emacs 30.0.93 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.43, cairo version 1.18.2) |
| git | git version 2.47.1 |
Output
Warning (elpaca): Unable to determine elpaca-core-date
INFO Scraping 16 files for loaddefs...
INFO Scraping 16 files for loaddefs...done
GEN ../elpaca-autoloads.el
Cloning into '/tmp/elpaca.vLbhmd/elpaca/repos/elpaca'...
Note: switching to '722b36c'.
You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.
If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:
git switch -c <new-branch-name>
Or undo this operation with:
git switch -
Turn off this advice by setting config variable advice.detachedHead to false
HEAD is now at 722b36c add basic lock file support
Checking /tmp/elpaca.vLbhmd/elpaca/repos/elpaca...
Compiling /tmp/elpaca.vLbhmd/elpaca/repos/elpaca/elpaca-info.el...
Warning (elpaca): Unable to determine elpaca-core-date
Compiling /tmp/elpaca.vLbhmd/elpaca/repos/elpaca/elpaca-log.el...
Compiling /tmp/elpaca.vLbhmd/elpaca/repos/elpaca/elpaca-manager.el...
Compiling /tmp/elpaca.vLbhmd/elpaca/repos/elpaca/elpaca-menu-elpa.el...
Compiling /tmp/elpaca.vLbhmd/elpaca/repos/elpaca/elpaca-menu-melpa.el...
Compiling /tmp/elpaca.vLbhmd/elpaca/repos/elpaca/elpaca-menu-org.el...
Compiling /tmp/elpaca.vLbhmd/elpaca/repos/elpaca/elpaca-process.el...
Compiling /tmp/elpaca.vLbhmd/elpaca/repos/elpaca/elpaca-test.el...
Compiling /tmp/elpaca.vLbhmd/elpaca/repos/elpaca/elpaca-ui.el...
Compiling /tmp/elpaca.vLbhmd/elpaca/repos/elpaca/elpaca.el...
Checking /tmp/elpaca.vLbhmd/elpaca/repos/elpaca/doc...
Compiling /tmp/elpaca.vLbhmd/elpaca/repos/elpaca/doc/early-init.el...
Compiling /tmp/elpaca.vLbhmd/elpaca/repos/elpaca/doc/init.el...
Compiling /tmp/elpaca.vLbhmd/elpaca/repos/elpaca/doc/installer.el...
Checking /tmp/elpaca.vLbhmd/elpaca/repos/elpaca/extensions...
Compiling /tmp/elpaca.vLbhmd/elpaca/repos/elpaca/extensions/elpaca-use-package.el...
Checking /tmp/elpaca.vLbhmd/elpaca/repos/elpaca/images...
Checking /tmp/elpaca.vLbhmd/elpaca/repos/elpaca/test...
Compiling /tmp/elpaca.vLbhmd/elpaca/repos/elpaca/test/elpaca-test.el...
Compiling /tmp/elpaca.vLbhmd/elpaca/repos/elpaca/test/elpaca-tests.el...
Done (Total of 12 files compiled, 4 skipped in 4 directories)
Downloading MELPA recipes...
Downloading MELPA recipes...100%
elpaca checking-out-ref Checking out 722b36c 02.988554
elpaca checking-out-ref $git -c advice.detachedHead=false checkout 722b36c 02.988850
general cloning Cloning 02.989457
general cloning $git clone --filter=tree:0 https://github.com/noctuid/general.el.git /tmp/elpaca.vLbhmd/elpaca/repos/general/ 02.989994
elpaca checking-out-ref HEAD is now at 722b36c add basic lock file support 02.993697
general cloning Cloning into '/tmp/elpaca.vLbhmd/elpaca/repos/general'... 02.994303
elpaca ref-checked-out 722b36c checked out 02.994681
elpaca unblocked No external dependencies 02.995436
elpaca unblocked Checking dependency versions 02.995781
elpaca linking Linking build files 02.996237
elpaca linking Build files linked 02.998375
elpaca autoloads Generating autoloads: /tmp/elpaca.vLbhmd/elpaca/builds/elpaca 02.998769
elpaca autoloads $/nix/store/9hc1gja85364wz5r4zvnl2i3741sjaak-emacs-pgtk-30.0.93/bin/emacs-30.0.93 -Q -L /tmp/elpaca.vLbhmd/elpaca/repos/elpaca/ -l /tmp/elpaca.vLbhmd/elpaca/repos/elpaca/elpaca.el --batch --eval (progn (setq gc-cons-percentage 1.0) (elpaca-generate-autoloads "elpaca" "/tmp/elpaca.vLbhmd/elpaca/builds/elpaca")) 02.999208
elpaca autoloads Warning (elpaca): Unable to determine elpaca-core-date 03.092302
elpaca autoloads INFO Scraping 11 files for loaddefs... 03.111292
elpaca autoloads INFO Scraping 11 files for loaddefs...done 03.124415
elpaca autoloads GEN elpaca-autoloads.el 03.147893
elpaca autoloads Autoloads Generated 03.151696
elpaca byte-compilation Byte compiling 03.152142
elpaca byte-compilation $/nix/store/9hc1gja85364wz5r4zvnl2i3741sjaak-emacs-pgtk-30.0.93/bin/emacs-30.0.93 -q --batch --eval (let ((gc-cons-percentage 1.0) (native-comp-eln-load-path '("/home/alan/.emacs.d/eln-cache/" "/nix/store/9hc1gja85364wz5r4zvnl2i3741sjaak-emacs-pgtk-30.0.93/lib/emacs/30.0.93/native-lisp/"))) (dolist (dir '("/tmp/elpaca.vLbhmd/elpaca/builds/elpaca")) (let ((default-directory dir)) (add-to-list 'load-path dir) (normal-top-level-add-subdirs-to-load-path))) (byte-recompile-directory "/tmp/elpaca.vLbhmd/elpaca/builds/elpaca" 0 'force)) 03.152524
elpaca byte-compilation Checking /tmp/elpaca.vLbhmd/elpaca/builds/elpaca... 03.232411
elpaca byte-compilation Compiling /tmp/elpaca.vLbhmd/elpaca/builds/elpaca/elpaca-autoloads.el... 03.239068
elpaca byte-compilation Compiling /tmp/elpaca.vLbhmd/elpaca/builds/elpaca/elpaca-info.el... 03.239793
general cloning remote: Enumerating objects: 309, done. 03.265276
elpaca byte-compilation Warning (elpaca): Unable to determine elpaca-core-date 03.275953
general cloning remote: Counting objects: 100% (309/309), done. 03.276704
general cloning remote: Compressing objects: 100% (307/307), done. 03.289792
general cloning remote: Total 309 (delta 2), reused 302 (delta 2), pack-reused 0 (from 0) 03.330440
general cloning Receiving objects: 100% (309/309), 87.12 KiB | 2.72 MiB/s, done. 03.331004
general cloning Resolving deltas: 100% (2/2), done. 03.331453
elpaca byte-compilation Compiling /tmp/elpaca.vLbhmd/elpaca/builds/elpaca/elpaca-log.el... 03.334919
elpaca byte-compilation Compiling /tmp/elpaca.vLbhmd/elpaca/builds/elpaca/elpaca-manager.el... 03.345640
elpaca byte-compilation Compiling /tmp/elpaca.vLbhmd/elpaca/builds/elpaca/elpaca-menu-elpa.el... 03.350678
elpaca byte-compilation Compiling /tmp/elpaca.vLbhmd/elpaca/builds/elpaca/elpaca-menu-melpa.el... 03.366584
elpaca byte-compilation Compiling /tmp/elpaca.vLbhmd/elpaca/builds/elpaca/elpaca-menu-org.el... 03.371420
elpaca byte-compilation Compiling /tmp/elpaca.vLbhmd/elpaca/builds/elpaca/elpaca-process.el... 03.381439
elpaca byte-compilation Compiling /tmp/elpaca.vLbhmd/elpaca/builds/elpaca/elpaca-test.el... 03.381948
elpaca byte-compilation Compiling /tmp/elpaca.vLbhmd/elpaca/builds/elpaca/elpaca-ui.el... 03.391574
elpaca byte-compilation Compiling /tmp/elpaca.vLbhmd/elpaca/builds/elpaca/elpaca.el... 03.403366
elpaca byte-compilation Done (Total of 10 files compiled, 1 skipped) 03.497048
elpaca byte-compilation Byte compilation complete 03.500049
elpaca info Compiling Info files 03.500601
elpaca info makeinfo not found 03.501070
elpaca info Installing Info files 03.501529
elpaca info No Info dir file found 03.502556
elpaca finished ✓ 3.438 secs 03.503020
general cloning remote: Enumerating objects: 4, done. 03.597244
general cloning remote: Counting objects: 100% (4/4), done. 03.598146
general cloning remote: Compressing objects: 100% (2/2), done. 03.598753
general cloning remote: Total 4 (delta 0), reused 2 (delta 0), pack-reused 0 (from 0) 03.599349
general cloning Receiving objects: 100% (4/4), 727 bytes | 727.00 KiB/s, done. 03.610283
general cloning remote: Enumerating objects: 14, done. 03.836606
general cloning remote: Counting objects: 100% (14/14), done. 03.837811
general cloning remote: Compressing objects: 100% (13/13), done. 03.849000
general cloning remote: Total 14 (delta 1), reused 9 (delta 0), pack-reused 0 (from 0) 03.896983
general cloning Receiving objects: 100% (14/14), 85.92 KiB | 2.15 MiB/s, done. 03.900399
general cloning Resolving deltas: 100% (1/1), done. 03.900961
general unblocked No external dependencies 03.943838
general unblocked Checking dependency versions 03.944328
general linking Linking build files 03.944808
general linking Build files linked 03.946037
general autoloads Generating autoloads: /tmp/elpaca.vLbhmd/elpaca/builds/general 03.946553
general autoloads $/nix/store/9hc1gja85364wz5r4zvnl2i3741sjaak-emacs-pgtk-30.0.93/bin/emacs-30.0.93 -Q -L /tmp/elpaca.vLbhmd/elpaca/repos/elpaca/ -l /tmp/elpaca.vLbhmd/elpaca/repos/elpaca/elpaca.el --batch --eval (progn (setq gc-cons-percentage 1.0) (elpaca-generate-autoloads "general" "/tmp/elpaca.vLbhmd/elpaca/builds/general")) 03.947044
general autoloads Warning (elpaca): Unable to determine elpaca-core-date 04.033050
general autoloads INFO Scraping 1 files for loaddefs... 04.050876
general autoloads INFO Scraping 1 files for loaddefs...done 04.063002
general autoloads GEN general-autoloads.el 04.067973
general autoloads Autoloads Generated 04.068941
general byte-compilation Byte compiling 04.069735
general byte-compilation $/nix/store/9hc1gja85364wz5r4zvnl2i3741sjaak-emacs-pgtk-30.0.93/bin/emacs-30.0.93 -q --batch --eval (let ((gc-cons-percentage 1.0) (native-comp-eln-load-path '("/home/alan/.emacs.d/eln-cache/" "/nix/store/9hc1gja85364wz5r4zvnl2i3741sjaak-emacs-pgtk-30.0.93/lib/emacs/30.0.93/native-lisp/"))) (dolist (dir '("/tmp/elpaca.vLbhmd/elpaca/builds/general")) (let ((default-directory dir)) (add-to-list 'load-path dir) (normal-top-level-add-subdirs-to-load-path))) (byte-recompile-directory "/tmp/elpaca.vLbhmd/elpaca/builds/general" 0 'force)) 04.070292
general byte-compilation Checking /tmp/elpaca.vLbhmd/elpaca/builds/general... 04.147375
general byte-compilation Compiling /tmp/elpaca.vLbhmd/elpaca/builds/general/general-autoloads.el... 04.153675
general byte-compilation Compiling /tmp/elpaca.vLbhmd/elpaca/builds/general/general.el... 04.154382
general byte-compilation In toplevel form: 04.167956
general byte-compilation general.el:63:12: Warning: custom-declare-variable ‘general-default-non-normal-prefix’ docstring has wrong usage of unescaped single quotes (use \=' or different quoting such as `...') 04.168805
general byte-compilation general.el:138:12: Warning: custom-declare-variable ‘general-vim-definer-default’ docstring has wrong usage of unescaped single quotes (use \=' or different quoting such as `...') 04.173874
general byte-compilation general.el:162:28: Warning: in define-widget for ‘general-alist’: ‘or’ is not a valid type 04.174503
general byte-compilation general.el:235:12: Warning: custom-declare-variable ‘general-describe-update-previous-definition’ docstring has wrong usage of unescaped single quotes (use \=' or different quoting such as `...') 04.175053
general byte-compilation In general--get-keymap: 04.184674
general byte-compilation general.el:530:2: Warning: docstring has wrong usage of unescaped single quotes (use \=' or different quoting such as `...') 04.185857
general byte-compilation In general--parse-def: 04.195998
general byte-compilation general.el:948:2: Warning: docstring wider than 80 characters 04.196654
general byte-compilation In general-define-key: 04.209103
general byte-compilation general.el:1074:2: Warning: docstring has wrong usage of unescaped single quotes (use \=' or different quoting such as `...') 04.209945
general byte-compilation In general-unbind: 04.214548
general byte-compilation general.el:1414:2: Warning: docstring has wrong usage of unescaped single quotes (use \=' or different quoting such as `...') 04.215388
general byte-compilation In general-key: 04.224759
general byte-compilation general.el:1648:2: Warning: docstring has wrong usage of unescaped single quotes (use \=' or different quoting such as `...') 04.225519
general byte-compilation In general-predicate-dispatch: 04.234843
general byte-compilation general.el:2164:4: Warning: More than one doc string 04.235551
general byte-compilation In general-translate-key: 04.236173
general byte-compilation general.el:2186:2: Warning: docstring has wrong usage of unescaped single quotes (use \=' or different quoting such as `...') 04.236767
general byte-compilation In general--define-transient-function: 04.237371
general byte-compilation general.el:2335:2: Warning: docstring has wrong usage of unescaped single quotes (use \=' or different quoting such as `...') 04.237959
general byte-compilation In general-add-hook: 04.238571
general byte-compilation general.el:2388:2: Warning: docstring has wrong usage of unescaped single quotes (use \=' or different quoting such as `...') 04.239255
general byte-compilation In general-advice-add: 04.239952
general byte-compilation general.el:2417:2: Warning: docstring has wrong usage of unescaped single quotes (use \=' or different quoting such as `...') 04.240545
general byte-compilation In general-evil-setup: 04.241207
general byte-compilation general.el: Warning: Doc string after ‘declare’ 04.241826
general byte-compilation general.el: Warning: Doc string after ‘declare’ 04.245071
general byte-compilation general.el: Warning: Doc string after ‘declare’ 04.246086
general byte-compilation general.el: Warning: Doc string after ‘declare’ 04.246832
general byte-compilation general.el: Warning: Doc string after ‘declare’ 04.247687
general byte-compilation general.el: Warning: Doc string after ‘declare’ 04.248323
general byte-compilation general.el: Warning: Doc string after ‘declare’ 04.248985
general byte-compilation general.el: Warning: Doc string after ‘declare’ 04.249778
general byte-compilation general.el: Warning: Doc string after ‘declare’ 04.250428
general byte-compilation general.el: Warning: Doc string after ‘declare’ 04.251120
general byte-compilation general.el: Warning: Doc string after ‘declare’ 04.251800
general byte-compilation general.el: Warning: Doc string after ‘declare’ 04.252444
general byte-compilation general.el:2605:10: Warning: docstring has wrong usage of unescaped single quotes (use \=' or different quoting such as `...') 04.253088
general byte-compilation general.el:2605:10: Warning: docstring has wrong usage of unescaped single quotes (use \=' or different quoting such as `...') 04.253824
general byte-compilation In end of data: 04.254600
general byte-compilation general.el:2757:44: Warning: the function ‘use-package-handler/:ghook’ is not known to be defined. 04.255310
general byte-compilation general.el:2718:45: Warning: the function ‘general-normalize-hook’ is not known to be defined. 04.256012
general byte-compilation general.el:2715:25: Warning: the function ‘general-normalize-hook-arglist’ is not known to be defined. 04.256815
general byte-compilation general.el:2654:52: Warning: the function ‘use-package-handler/:general’ is not known to be defined. 04.257493
general byte-compilation general.el:2653:7: Warning: the function ‘use-package-normalize/:general’ is not known to be defined. 04.258169
general byte-compilation general.el:2610:33: Warning: the function ‘general--sanitize-arglist’ is not known to be defined. 04.258851
general byte-compilation general.el:2546:24: Warning: the function ‘general-tomap’ is not known to be defined. 04.259759
general byte-compilation general.el:2545:25: Warning: the function ‘general-otomap’ is not known to be defined. 04.260532
general byte-compilation general.el:2544:25: Warning: the function ‘general-itomap’ is not known to be defined. 04.261255
general byte-compilation general.el:2543:24: Warning: the function ‘general-nvmap’ is not known to be defined. 04.262055
general byte-compilation general.el:2542:24: Warning: the function ‘general-iemap’ is not known to be defined. 04.262762
general byte-compilation general.el:2541:23: Warning: the function ‘general-rmap’ is not known to be defined. 04.263563
general byte-compilation general.el:2540:23: Warning: the function ‘general-omap’ is not known to be defined. 04.264279
general byte-compilation general.el:2539:23: Warning: the function ‘general-mmap’ is not known to be defined. 04.265034
general byte-compilation general.el:2538:23: Warning: the function ‘general-vmap’ is not known to be defined. 04.265747
general byte-compilation general.el:2537:23: Warning: the function ‘general-nmap’ is not known to be defined. 04.266468
general byte-compilation general.el:2536:23: Warning: the function ‘general-emap’ is not known to be defined. 04.267185
general byte-compilation general.el:2535:23: Warning: the function ‘general-imap’ is not known to be defined. 04.267910
general byte-compilation Done (Total of 1 file compiled, 1 skipped) 04.268643
general byte-compilation Byte compilation complete 04.269438
general info Compiling Info files 04.270341
general info makeinfo not found 04.271070
general info Installing Info files 04.271824
general info No Info dir file found 04.273304
general activation Activating package 04.274052
general activation Package build dir added to load-path 04.274828
general activation Caching autoloads 04.275592
general activation Autoloads cached 04.276606
general finished ✓ 1.292 secs 04.277657
I've made some progress here. Below is a test demo of the ideas I mentioned in my previous comments.
On the "feat/refactor" branch the :build recipe keyword has been revamped to include a DSL which allows for substitution of specific steps. There is an elpaca-defstep macro which will return a function that executes the BODY forms in an Emacs subprocess. The body is implicitly backquoted, so injecting state from the parent process is done via the normal unquoting syntax. See the following test case:
Test Case
(elpaca-test
:ref "feat/refactor"
:installer
"https://raw.githubusercontent.com/progfolio/elpaca/refs/heads/feat/refactor/doc/installer.el"
:early-init
(setq elpaca-menu-functions '(elpaca-menu-extensions elpaca-menu-declarations))
:init
;; This value is injected into the child process below
(setq gc-cons-threshold 10000000)
(elpaca `(miscellany
:wait t
:host github
:repo ("progfolio/miscellany.el" . "miscellany")
:build (:sub
(:before elpaca--queue-dependencies
;; Define a build step which tangles an Org file to the main elisp file.
,(elpaca-defstep +miscellany-tangle
;;This BODY is executed in the Emacs subprocess.
(require 'ob-tangle)
;; `gc-cons-threshold' is injected from the parent process.
(message "injected state: %S" ,gc-cons-threshold)
(setq org-confirm-babel-evaluate nil)
(org-babel-tangle-file "./miscellany.org"))
;; This is an example of manually creating a build step which is run
;; synchronously in the main Emacs process.
(lambda (e)
(elpaca-with-dir e source
(elpaca--call-with-log e 0 "git" "log" "--oneline" "-n" "1")
(elpaca--continue-build e)))))))
(princ (elpaca-log "miscellany")))
Host Env
| elpaca | e9ab3f6 HEAD -> feat/refactor, origin/feat/refactor |
| installer | 0.11 |
| emacs | GNU Emacs 31.0.50 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.50, cairo version 1.18.4) of 2025-09-12 |
| git | git version 2.51.0 |
Output
INFO Scraping 17 files for loaddefs...
INFO Scraping 17 files for loaddefs...done
GEN ../elpaca-autoloads.el
Cloning into '/tmp/elpaca.RhGEWd/elpaca/repos/elpaca'...
Switched to a new branch 'feat/refactor'
branch 'feat/refactor' set up to track 'origin/feat/refactor'.
Checking /tmp/elpaca.RhGEWd/elpaca/repos/elpaca...
Compiling /tmp/elpaca.RhGEWd/elpaca/repos/elpaca/elpaca-git.el...
Compiling /tmp/elpaca.RhGEWd/elpaca/repos/elpaca/elpaca-info.el...
Compiling /tmp/elpaca.RhGEWd/elpaca/repos/elpaca/elpaca-local.el...
Compiling /tmp/elpaca.RhGEWd/elpaca/repos/elpaca/elpaca-log.el...
Compiling /tmp/elpaca.RhGEWd/elpaca/repos/elpaca/elpaca-manager.el...
Compiling /tmp/elpaca.RhGEWd/elpaca/repos/elpaca/elpaca-menu-elpa.el...
Compiling /tmp/elpaca.RhGEWd/elpaca/repos/elpaca/elpaca-menu-melpa.el...
Compiling /tmp/elpaca.RhGEWd/elpaca/repos/elpaca/elpaca-menu-org.el...
Compiling /tmp/elpaca.RhGEWd/elpaca/repos/elpaca/elpaca-process.el...
Compiling /tmp/elpaca.RhGEWd/elpaca/repos/elpaca/elpaca-test.el...
Compiling /tmp/elpaca.RhGEWd/elpaca/repos/elpaca/elpaca-ui.el...
Compiling /tmp/elpaca.RhGEWd/elpaca/repos/elpaca/elpaca.el...
Checking /tmp/elpaca.RhGEWd/elpaca/repos/elpaca/doc...
Compiling /tmp/elpaca.RhGEWd/elpaca/repos/elpaca/doc/early-init.el...
Compiling /tmp/elpaca.RhGEWd/elpaca/repos/elpaca/doc/init.el...
Compiling /tmp/elpaca.RhGEWd/elpaca/repos/elpaca/doc/installer.el...
Checking /tmp/elpaca.RhGEWd/elpaca/repos/elpaca/extensions...
Compiling /tmp/elpaca.RhGEWd/elpaca/repos/elpaca/extensions/elpaca-use-package.el...
Checking /tmp/elpaca.RhGEWd/elpaca/repos/elpaca/images...
Checking /tmp/elpaca.RhGEWd/elpaca/repos/elpaca/test...
Compiling /tmp/elpaca.RhGEWd/elpaca/repos/elpaca/test/elpaca-tests.el...
Done (Total of 14 files compiled, 3 skipped in 4 directories)
miscellany cloning Cloning 00.076839
miscellany cloning $git clone --filter=tree:0 https://github.com/progfolio/miscellany.el.git /tmp/elpaca.RhGEWd/elpaca/repos/miscellany/ 00.077182
miscellany cloning Cloning into '/tmp/elpaca.RhGEWd/elpaca/repos/miscellany'... 00.079038
miscellany cloning remote: Enumerating objects: 23, done. 00.562136
miscellany cloning remote: Counting objects: 100% (23/23), done. 00.563185
miscellany cloning remote: Compressing objects: 100% (23/23), done. 00.563776
miscellany cloning remote: Total 23 (delta 0), reused 23 (delta 0), pack-reused 0 (from 0) 00.564415
miscellany cloning Receiving objects: 100% (23/23), 4.08 KiB | 4.08 MiB/s, done. 00.565212
miscellany cloning remote: Enumerating objects: 1, done. 00.762935
miscellany cloning remote: Counting objects: 100% (1/1), done. 00.763554
miscellany cloning remote: Total 1 (delta 0), reused 0 (delta 0), pack-reused 0 (from 0) 00.763938
miscellany cloning Receiving objects: 100% (1/1), 151 bytes | 151.00 KiB/s, done. 00.764952
miscellany cloning remote: Enumerating objects: 3, done. 00.965069
miscellany cloning remote: Counting objects: 100% (3/3), done. 00.965675
miscellany cloning remote: Compressing objects: 100% (2/2), done. 00.966122
miscellany cloning remote: Total 3 (delta 1), reused 1 (delta 0), pack-reused 0 (from 0) 00.967099
miscellany cloning Receiving objects: 100% (3/3), 5.32 KiB | 5.32 MiB/s, done. 00.968573
miscellany cloning Resolving deltas: 100% (1/1), done. 00.969023
miscellany ref-checked-out $/usr/bin/emacs -Q --batch --eval (progn (require 'ob-tangle) (message "injected state: %S" 10000000) (setq org-confirm-babel-evaluate nil) (org-babel-tangle-file "./miscellany.org")) 00.982058
miscellany ref-checked-out injected state: 10000000 01.076627
miscellany ref-checked-out Tangled 1 code block from miscellany.org 01.319242
miscellany ref-checked-out elpaca-with-emacs-miscellany complete 01.329242
miscellany ref-checked-out $git log --oneline -n 1 01.331708
miscellany ref-checked-out 54867f6 toggle-maximize-buffer: guard against maximizing minibuffer 01.332378
miscellany unblocked No external dependencies 01.332962
miscellany unblocked Checking dependency versions 01.333418
miscellany linking Linking build files 01.333796
miscellany linking Build files linked 01.334609
miscellany autoloads Generating autoloads: /tmp/elpaca.RhGEWd/elpaca/builds/miscellany 01.335025
miscellany autoloads $/usr/bin/emacs -Q --batch -L /tmp/elpaca.RhGEWd/elpaca/repos/elpaca/ -l /tmp/elpaca.RhGEWd/elpaca/repos/elpaca/elpaca.el --eval (let (print-level print-circle) (setq gc-cons-percentage 1.0) (elpaca-generate-autoloads "miscellany" "/tmp/elpaca.RhGEWd/elpaca/builds/miscellany")) 01.335418
miscellany autoloads INFO Scraping 1 files for loaddefs... 01.432929
miscellany autoloads INFO Scraping 1 files for loaddefs...done 01.434099
miscellany autoloads GEN miscellany-autoloads.el 01.434605
miscellany autoloads autoloads complete 01.439791
miscellany byte-compilation Byte compiling 01.440330
miscellany byte-compilation $/usr/bin/emacs -Q --batch --eval (progn (setq gc-cons-percentage 1.0 print-level nil print-circle nil native-comp-eln-load-path '("/home/n/.emacs.d/eln-cache/" "/usr/lib/emacs/31.0.50/native-lisp/")) (dolist (dir '("/tmp/elpaca.RhGEWd/elpaca/builds/miscellany")) (let ((default-directory dir)) (add-to-list 'load-path dir) (normal-top-level-add-subdirs-to-load-path))) (byte-recompile-directory "/tmp/elpaca.RhGEWd/elpaca/builds/miscellany" 0 'force)) 01.441256
miscellany byte-compilation Checking /tmp/elpaca.RhGEWd/elpaca/builds/miscellany... 01.510702
miscellany byte-compilation Compiling /tmp/elpaca.RhGEWd/elpaca/builds/miscellany/miscellany-autoloads.el... 01.512180
miscellany byte-compilation Compiling /tmp/elpaca.RhGEWd/elpaca/builds/miscellany/miscellany.el... 01.512819
miscellany byte-compilation In +compute-blood-pressure-table-row: 01.530647
miscellany byte-compilation miscellany.el:57:4: Warning: ‘if-let’ is an obsolete macro (as of 31.1); use ‘if-let*’ instead. 01.531652
miscellany byte-compilation In +screenshot: 01.532516
miscellany byte-compilation miscellany.el:268:4: Warning: ‘when-let’ is an obsolete macro (as of 31.1); use ‘when-let*’ or ‘and-let*’ instead. 01.533009
miscellany byte-compilation Done (Total of 1 file compiled, 1 skipped) 01.533480
miscellany byte-compilation Byte compilation complete 01.535728
miscellany info Compiling Info files 01.536190
miscellany info Info source files not found 01.536657
miscellany activation Activating package 01.537100
miscellany activation Package build dir added to load-path 01.537549
miscellany activation Caching autoloads 01.538026
miscellany activation Autoloads cached 01.538622
miscellany finished ✓ 1.483 secs 01.539091
This is much more flexible than the :pre-build and :post-build keywords, but still needs some testing.
I plan on dog-fooding this for a bit and merging soon.
Here's a second draft which cleans up the :build DSL and allows elpaca-defstep to do both direct asynchronous processes and an Emacs subprocess depending on the args passed.
In consideration of #414, the process exit code is respected in both cases as well.
Test Case
(elpaca-test
:ref "feat/refactor"
:installer
"https://raw.githubusercontent.com/progfolio/elpaca/refs/heads/feat/refactor/doc/installer.el"
:early-init
(setq elpaca-menu-functions '(elpaca-menu-declarations))
:init
;; This value is injected into the child process below
(setq gc-cons-threshold 10000000)
(elpaca `(miscellany
:wait t
:host github
:repo ("progfolio/miscellany.el" . "miscellany")
:build (:before elpaca--queue-dependencies
;; Define a build step which tangles an Org file to the main elisp file.
,(elpaca-defstep +miscellany-tangle
;;This BODY is executed in the Emacs subprocess.
(require 'ob-tangle)
;; `gc-cons-threshold' is injected from the parent process.
(message "injected state: %S" ,gc-cons-threshold)
(setq org-confirm-babel-evaluate nil)
(org-babel-tangle-file "./miscellany.org"))
;; Asynchronously run the following git command...
,(elpaca-defstep +miscellany-log-head
("git" "log" "--oneline" "-n" "1")))))
(princ (elpaca-log "miscellany | script")))
Host Env
| elpaca | 8f836b9 HEAD -> feat/refactor, origin/feat/refactor |
| installer | 0.11 |
| emacs | GNU Emacs 31.0.50 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.50, cairo version 1.18.4) of 2025-09-12 |
| git | git version 2.51.0 |
Output
INFO Scraping 17 files for loaddefs...
INFO Scraping 17 files for loaddefs...done
GEN ../elpaca-autoloads.el
Cloning into '/tmp/elpaca.Y3IXdd/elpaca/repos/elpaca'...
Switched to a new branch 'feat/refactor'
branch 'feat/refactor' set up to track 'origin/feat/refactor'.
Checking /tmp/elpaca.Y3IXdd/elpaca/repos/elpaca...
Compiling /tmp/elpaca.Y3IXdd/elpaca/repos/elpaca/elpaca-git.el...
Compiling /tmp/elpaca.Y3IXdd/elpaca/repos/elpaca/elpaca-info.el...
Compiling /tmp/elpaca.Y3IXdd/elpaca/repos/elpaca/elpaca-local.el...
Compiling /tmp/elpaca.Y3IXdd/elpaca/repos/elpaca/elpaca-log.el...
Compiling /tmp/elpaca.Y3IXdd/elpaca/repos/elpaca/elpaca-manager.el...
Compiling /tmp/elpaca.Y3IXdd/elpaca/repos/elpaca/elpaca-menu-elpa.el...
Compiling /tmp/elpaca.Y3IXdd/elpaca/repos/elpaca/elpaca-menu-melpa.el...
Compiling /tmp/elpaca.Y3IXdd/elpaca/repos/elpaca/elpaca-menu-org.el...
Compiling /tmp/elpaca.Y3IXdd/elpaca/repos/elpaca/elpaca-process.el...
Compiling /tmp/elpaca.Y3IXdd/elpaca/repos/elpaca/elpaca-test.el...
Compiling /tmp/elpaca.Y3IXdd/elpaca/repos/elpaca/elpaca-ui.el...
Compiling /tmp/elpaca.Y3IXdd/elpaca/repos/elpaca/elpaca.el...
Checking /tmp/elpaca.Y3IXdd/elpaca/repos/elpaca/doc...
Compiling /tmp/elpaca.Y3IXdd/elpaca/repos/elpaca/doc/early-init.el...
Compiling /tmp/elpaca.Y3IXdd/elpaca/repos/elpaca/doc/init.el...
Compiling /tmp/elpaca.Y3IXdd/elpaca/repos/elpaca/doc/installer.el...
Checking /tmp/elpaca.Y3IXdd/elpaca/repos/elpaca/extensions...
Compiling /tmp/elpaca.Y3IXdd/elpaca/repos/elpaca/extensions/elpaca-use-package.el...
Checking /tmp/elpaca.Y3IXdd/elpaca/repos/elpaca/images...
Checking /tmp/elpaca.Y3IXdd/elpaca/repos/elpaca/test...
Compiling /tmp/elpaca.Y3IXdd/elpaca/repos/elpaca/test/elpaca-tests.el...
Done (Total of 14 files compiled, 3 skipped in 4 directories)
miscellany script +miscellany-tangle 00.808548
miscellany script $/usr/bin/emacs -Q --batch --eval (setq gc-cons-percentage 1.0 print-level nil print-circle nil) --eval (progn nil (require 'ob-tangle) (message "injected state: %S" 10000000) (setq org-confirm-babel-evaluate nil) (org-babel-tangle-file "./miscellany.org")) 00.808966
miscellany script injected state: 10000000 00.884099
miscellany script Tangled 1 code block from miscellany.org 01.078661
miscellany script elpaca-with-emacs-miscellany complete 01.085746
miscellany script +miscellany-log-head-1 01.086300
miscellany script $git log --oneline -n 1 01.086684
miscellany script 54867f6 toggle-maximize-buffer: guard against maximizing minibuffer 01.088643