epm packages depending on external commands

[xiaq]

As a shell, Elvish encourages taking advantage of external commands. As such, EPM packages may depend on external commands. Such dependencies currently have to be documented by the package authors, and the users will have to remember installing them. The EPM system could provide some help to reduce the friction.

There are roughly 3 different scenarios when a package depends on external commands:

1. The package depends on external commands typically provided by a UNIX system, e.g. cat and grep. The same command can have different flags and different behaviors across UNIX flavors, and are often non-existent on Windows.

   Elvish should help package authors handle the platform differences. One possible approach is to provide a builtin module exposing a "least common denominator" version of these commands. The package authors will need to remember to use that module instead of using the commands directly though. Maybe this is a valid use case to support "import from" found in other languages like Python and JavaScript.

2. The package depends on external commands that are not part of UNIX systems, but popular enough to be packaged by multiple system-level package managers, e.g. ffmpeg, gm and pandoc. EPM can help by letting the package author specify the following:

   • How to install the dependencies with different package managers
   • The required version range

   Automatically running the installation instructions and checking the version can be very useful.

3. The package either depends on external commands that are not popular enough to be packaged by package managers, or depends on a version that is too new to be packaged. In this case, the command will have to be built. EPM can help in two ways:

   • Accept building instructions and build the dependencies on the user's computer, or
   • Accept pre-built binaries provided by the package author

   This is in fact somewhat analogous to (2) and may overlap with it.

   A key concern here is that the external command (whether built on user's computer or pre-built) may be placed somewhere specific to the EPM package (and outside the default PATH) since it's less likely to be generally useful to the user's system.

These are just some preliminary thoughts. I might split these into separate issues if the implementation for different pieces turn out to be disjoint enough.

[krader1961]

As a shell, Elvish encourages taking advantage of external commands.

Yes, but that's pretty much true of every command shell in the traditional UNIX/Windows sense. 😺

Implementing option one ("least common denomator" version) is fraught. See, for example, the discussion about a builtin equivalent of realpath. Also, given that you have recently expressed a strong desire to keep the size of the Elvish binary as small as possible it seems to me this option to the problem is inconsistent with that desire.

I get the sense you are proposing an enhancement to the epm package that would allow an Elvish package owner to document external command dependencies. Such that those dependencies could be verified, even if only to verify an external command of the required name can be resolved (e.g., as by has-external). I admit, however, it is not clear to me what you actually have in mind with respect to this issue.

[xiaq]

Implementing option one ("least common denomator" version) is fraught. See, for example, the discussion about a builtin equivalent of realpath. Also, given that you have recently expressed a strong desire to keep the size of the Elvish binary as small as possible it seems to me this option to the problem is inconsistent with that desire.

I should clarify that the proposal there is not to reimplement UNIX utils within Elvish but to provide wrappers that only support features that are commonly supported. Differences in behavior when given the same flags are hard to paper over of course.

[krader1961]

I should clarify that the proposal there is not to reimplement UNIX utils within Elvish but to provide wrappers that only support features that are commonly supported. Differences in behavior when given the same flags are hard to paper over of course.

I'm still unclear what is meant by "provide wrappers". If Elvish doesn't directly implement the command as a builtin how would it wrap (i.e., paper over the differences) of an external command in any portable manner? Especially since commands like cat and grep don't even exist on Windows; let alone dealing with the differences in flags between BSD/Solaris/Linux etcetera.

[fennewald]

A possible approach to (3) could be elvish bundling an existing wasm runtime (most likely wasmer. epm packagers could include a wasm binary directly in their module, and load and run it. Otherwise, platform indepdent code would be very hard to package.

This doesn't solve the general case though, as many external commands require interactions with the system more complex then covered wasi. It is also pretty heavy-weight.

[xiaq]

A possible approach to (3) could be elvish bundling an existing wasm runtime (most likely wasmer. epm packagers could include a wasm binary directly in their module, and load and run it. Otherwise, platform indepdent code would be very hard to package.

This doesn't solve the general case though, as many external commands require interactions with the system more complex then covered wasi. It is also pretty heavy-weight.

A full WASM runtime is almost certainly too big to be acceptable...

I'd rather just make it easy to download binaries. If you're writing something new today with a modern-ish language, chances are it's quite easy to build static binaries - I'm thinking Go of course, but Rust is also easy. If you have something in C or C++ you may even be able to use APE.

EPM can then support, say, declaring URLs for downloading the binaries with placeholders for $platform:os and $platform:arch. An example for Elvish itself would be https://dl.elv.sh/{os}-{arch}/elvish-HEAD.

[xiaq]

I should clarify that the proposal there is not to reimplement UNIX utils within Elvish but to provide wrappers that only support features that are commonly supported. Differences in behavior when given the same flags are hard to paper over of course.

I'm still unclear what is meant by "provide wrappers". If Elvish doesn't directly implement the command as a builtin how would it wrap (i.e., paper over the differences) of an external command in any portable manner? Especially since commands like cat and grep don't even exist on Windows; let alone dealing with the differences in flags between BSD/Solaris/Linux etcetera.

You can't depend on the existence of cat or grep on Windows, but there's still a need to write scripts that are portable across UNIX flavors and sometimes that needs to use features that behave slightly differently.

One example is how FreeBSD's sed -i (and by extension macOS's) differs from all other UNIX flavors when you're editing files in-place with no backup:

sed -i '' s/foo/bar/ a.txt # FreeBSD / macOS
sed -i s/foo/bar/ a.txt # all other UNIX flavors

A solution could be Elvish bundling the following definition in a module named, say unixutils:

var sed-i~
if (has-value [darwin freebsd] $platform:os) {
  set sed-i~ = {|@a| sed -i '' $@a }
} else {
  set sed-i~ = {|@a| sed -i $@a}
}

And module authors can use unixutils:sed-i s/foo/bar/ a.txt for a bit more portability.

The implementation is just for illustration - sed may not necessarily point to the system's original sed, so a more robust version should use some sort of feature detection technique instead of depending on $platform:os.

[krader1961]

You can't depend on the existence of cat or grep on Windows, but there's still a need to write scripts that are portable across UNIX flavors and sometimes that needs to use features that behave slightly differently.

Yes, and that is precisely why I have implemented path:stat which is pending as PR #1688. The stat command has different options depending on whether you are on a GNU or BSD flavor. And, of course, stat doesn't even exist as an external command on Windows unless you have installed something like Msys2.

I also think my proposal in issue #1681 to add external command callbacks that allow for substituting a different implementation, potentially using only Elvish builtins, is a more flexible solution to this problem.