How to write self-sufficient dispatch-entry scripts

[amerlyq]

Problem

1. Using --dispatch-entry isn't intuitive and in no way easy for simple scripts.
2. Necessity of .asd for self-compiled one-file scripts is discouraging.
3. Caching binaries is not automated.

Story

The most significant (at least for me) point of other scripting languages with shebang lays in their ability to provide self-sufficient one-file scripts. Such scripts may depend on libraries installed somewhere in distribution-specific ways or downloaded by user. And these libraries are mostly discovered through some runtime path environment variable -- with distribution-configured default value. Even C/C++ has caching wrapper to build and run scripts effortlessly. So most of the time it's a zero-configuration way.

On the other hand there are virtualenv/pip/gem/cargo/etc... which accomodate complex/frozen ecosystems for large projects. They are characterized by having some configuration file which describes everything. And they are too much pain to use outside their cozzy setup project directory. Notice how much it resembles current state of affairs with dispatch-entry compilation step which requires correctly configured environment and multiple .asd for everything to work perfectly.

And here we are with CL -- when we use cl-lauch to run simple scripts -- it works acceptably and at least on par with caching C/C++ solution -- by at least pre-compiling *.fsl. But when trying to go one step further and directly exec from lisp script into generated dump image -- it's easier to be said than to be done. Not talking about sharing part of the image between all of scripts in this case...

Usecase

What I actually expect to work -- assuming generated binaries will be stored in ~/.cache/common-lisp/bin-cache/:

#!/usr/bin/cl --cached
(defun main (argv)
  (format t "First~%~S~%" argv)
  t)

As you could notice, I don't even care about current package. If I planned to reuse some part of code between two scripts -- I would move it into separate ArchLinux AUR package under /usr/share/common-lisp/sources/<mycommon>/*

What this -C|--cached is actually means in my head is akin to

cl-launch --output "$HOME/.cache/common-lisp/bin-cache/$1" \
  --dump '!' --lisp sbcl -E main "$1" \
  && exec "$HOME/.cache/common-lisp/bin-cache/$1" || exit 1

Notice -- we will cache it under full path and exec after execution.

Proposal

Even better would be to reuse shared image automatically (if it's possible at all)

• read precompiled image (whichever was used for --output or with default path ~/.cache/common-lisp/bin-cache/default)
• augment current script with its own package/system and evaluate inside that image to append
• store image back into that path
• create symlink in ~/.cache/common-lisp/bin/ or ~/.local/bin/ for multibinary support pointing into that default image

Questions

1. Did I miss something in documentation and my expected proposal is already implemented ?
2. Are expected convention is actually quite different and I'm trying to do it the wrong way ?
3. How hard would it be to add into cl-launch so I could cease writing my own convinience scripts to auto-exec into multibinary image?

[fare]

Dear amerlyq,

I'm sorry I don't have time to look too much into detail this week or the next, but here are a few points:

• You don't need an .asd file to use cl-launch. cl-launch will implicitly create a system to load the files you ask it to load in the correct order.
• cl-launch relies on caching by ASDF to load all the fasls.
• Unhappily, ASDF can have large startup times, especially if you have large trees in your source-registry and don't use the source-registry-cache.
• You can already create and use images using --dump; unhappily it is not usually safe to build an image from a previous image, and even just checking whether the image is up-to-date can slow down startup to the point that there's then no clear advantage over fasls.
• More advanced systems with an ASDF daemon that builds necessary files from clean and invalidates out-of-date ones by watching the filesystem could help, but cannot be written portably, at least, not without much more machinery than I'm willing to come up with. Also, I'm not sure watching the filesystem can be made 100% reliable under load due to dropped events (or do you at least get a notification of dropped events so you can recheck everything from scratch?).
• See fare-scripts for how I build a multibinary image.

[amerlyq]

Thanks for your time spent on answering, fare!

Actually this question has come to mind after looking into fare-scripts and re-reading all posts and slides on scripting you once shared, so I was at least prepared.

But after tapping into the issue from different angles and re-reading your answers for N-th time I concluded that my previous proposal was controversal -- because it actually increases time between edit and run -- the second most significant characteristic which defines scripting itself for me. Therefore building images can't be integrated into normal development workflow seamlessly and will forever remain for the deployment/distribution phase only -- when exploratory work is actually done. It seems there is truly no way beside "simply" optimizing *.fasl load time by implementations.

Still, I wish we at least had option to build && run because currently this doesn't work:

cl-launch --output /tmp/exe --dump ! --entry main --file ./unix-main.lisp --execute