Toward an ideal distribution mechanism

[fosskers]

Hi guys.

Users have taken to the streets and are crying for a push-button way to:

• Declare Lisp and Java (Maven Central) dependencies side-by-side.
• Easily locate and load those Java dependencies into the REPL.
• Expose some main function from Lisp and bundle a single image / fatjar.
• Do all this from the comfort of an ABCL repl session.

Much of what’s required for this seemingly already exists. This thread elaborates prior art in the field, and lays out suggestions for future development.

Prior Art

abcl-asdf

    (asdf:defsystem #:abcl-telegram-bot
      :description "Create telegram bots with ABCL"
      :author "Alejandro Zamora Fonseca <[email protected]>"
      :license  "MIT"
      :version "0.0.1"
      :serial t
      :depends-on (:abcl-memory-compiler :alexandria)
      :components ((:mvn "org.telegram/telegrambots-longpolling/8.3.0")
                   (:mvn "org.telegram/telegrambots-client/8.3.0")
                   (:file "package")
                   (:file "abcl-telegram-bot")))

abcl-asdf extends ASDF to comprehend (:mvn ...) entries under :components. Upon system load, it downloads these Java dependencies to ~/.m2/, the location of which can’t be configured, even though the mvn CLI tool allows it:

mvn dependency:get \
    -DgroupId=org.apache.commons \
    -DartifactId=commons-text \
    -Dversion=1.13.1 \
    -Dmaven.repo.local=vendored/java/

Pros:

• This provides a first-class way to specify Java dependencies alongside Lisp ones.
• Downloading and classpath management is handled automatically.

Issues:

• Counter-intuitive is that these Java dependencies aren’t defined within :depends-on.
• If you haven’t already manually loaded abcl-contrib and abcl-asdf before loading this system, ASDF doesn’t know what to do with the :mvn blocks and throws an error.
• There is seemingly no way to tell abcl-asdf not to download or not to manage the classpath.

java:add-to-classpath

If you have a JAR somewhere on your machine, there is technically no need to go through abcl-asdf; adding it manually to the classpath is sufficient:

    (java:add-to-classpath "/home/colin/code/common-lisp/vend/vendored/java/commons-io/commons-io/2.16.1/commons-io-2.16.1.jar")
    (#"toAbsolutePath" (#"current" 'org.apache.commons.io.file.PathUtils))
    ;; => #<sun.nio.fs.UnixPath /home/colin/code/common-lisp/ven.... {440C8006}>

It’s simple enough for any tool trying to build up classpath entries to recursively parse POM-file XML and call add-to-classpath as appropriate.

One thing I’m not sure about is whether calling add-to-classpath is still necessary when we’ve already built a fatjar with jar cfm and a MANIFEST.MF. I suspect not but haven’t confirmed.

asdf-jar

This provides a way to package all our Lisp sources (and dependencies!) into a single JAR. Given:

    (defpackage abcl-test
      (:use :cl :arrow-macros)
      (:export #:launch))
    
    (in-package :abcl-test)
    
    (require :java)
    
    (defun launch ()
      (->> (java:jstatic "now" "java.time.LocalDate")
           (java:jcall "toString")
           (format t "Date: ~a~%")))

then by calling:

    (require :abcl-contrib)
    (require :asdf-jar)
    
    (asdf-jar:package :abcl-test :out #p"./" :fasls t :verbose t)

we get our JAR.

Issues:

• As mentioned in this issue, despite :fasls t, subsequent asdf:load-system calls don’t seem to respect the bundled .abcl fasl files.
• As mentiond in this issue, loading asdf-jar is inconsistent and often fails.

Lisp-from-Java wrapping and jar cfm

In theory a universal entry-point runner could be written on the Java side to run our program:

    import org.armedbear.lisp.Interpreter;
    
    public class Main
    {
        public static void main(String[] args) {
            Interpreter i = Interpreter.createInstance();
            i.eval("(require :asdf)");
            i.eval("(require :abcl-contrib)");
            i.eval("(require :asdf-jar)");
            // Somehow refer to the child JAR within this JAR.
            // i.eval("(asdf-jar:add-to-asdf \"/home/colin/code/common-lisp/abcl-test/abcl-test-all-0.0.1.jar\")");
            i.eval("(asdf:load-system :abcl-test)");
            i.eval("(abcl-test:launch)");
        }
    }

We can compile this with:

javac -cp /usr/share/java/abcl.jar Main.java

And given a MANIFST.MF:

Manifest-Version: 1.0
Main-Class: Main
Class-Path: /usr/share/java/abcl.jar /usr/share/java/abcl-contrib.jar /home/colin/code/common-lisp/abcl-test/abcl-test-all-0.0.1.jar

The Class-Path here is probably wrong, but all this can be bundled together into a single JAR with:

jar cfm app.jar MANIFEST.MF Main.class \
    -C /usr/share/java/ abcl.jar \
    -C /usr/share/java/ abcl-contrib.jar \
    -C /home/colin/code/common-lisp/abcl-test/ abcl-test-all-0.0.1.jar

We now have a “fatjar”. Then calling java -jar app.jar actually runs! But it currently always fails trying to load asdf-jar, as mentioned above. If we could get past that, and work out the classpath issues, in theory we have all the pieces (although not at all automated).

asdf:make

Briefly I will mention a few other solutions for inspiration.

asdf:make simply fails under ABCL. Given:

    (defsystem "abcl-test"
      :version "0.0.1"
      :depends-on (:arrow-macros)
      :components ((:module "src" :components ((:file "main"))))
      :build-operation program-op
      :build-pathname "abcl-test"
      :entry-point "abcl-test:launch")

We are told:

#<THREAD “interpreter” native {518FC5E}>: Debugger invoked on condition of type NOT-IMPLEMENTED-ERROR Not (currently) implemented on ABCL: UIOP/IMAGE:DUMP-IMAGE dumping an executable

(ECL) asdf:make-build

ECL relies on a special implementation of asdf:make-build, otherwise normally deprecated, for building its own binaries. This doesn’t require special entries in .asd. Here is how vend is built:

    (asdf:make-build :vend
                     :type :program
                     :move-here #p"./"
                     :epilogue-code '(vend:main))

Dead simple. See here for a bigger example that sets linker flags for binding to C (.so) dependencies.

(SBCL) sb-ext:save-lisp-and-die

SBCL doesn’t really differentiate between building an “image” and building an executable; in the latter case there is simply a well-defined entrypoint, and the blob can be run as-is from the terminal.

    (sb-ext:save-lisp-and-die #p"aero-fighter"
                              :toplevel #'aero-fighter:launch
                              :executable t
                              :compression t)

Note that you can’t call this from within Sly/Slime sessions, it typically must be done in a fresh, standalone REPL (or build script).

(Clojure) Naive running

Assuming the user has Clojure installed, it’s enough to run any program (with any mix of Clojure and Maven dependencies) by running commands like:

clojure -M -m some_namespace

Where there is a file in your project somewhere defining a namespace / module that contains a -main:

    (defn -main [& args]
      (println "Hi!"))

This “just works” with no extra config, and is especially convenient if you don’t need to “distribute” to non-devs.

(Clojure) lein uberjar

The Clojure tool lein is also able to build uberjars. With a separate project.clj:

    (defproject my-project "0.1.0-SNAPSHOT"
      :description "A Clojure project with Java dependencies"
      :dependencies [[org.clojure/clojure "1.11.1"]
                     [com.fasterxml.jackson.core/jackson-databind "2.17.2"]] ; Example Java dependency
      :main my-project.core
      :aot [my-project.core])

Then lein uberjar produces our fatjar that can be run with java -jar.

Recent Development

I have an experimental branch on vend that handles downloading through mvn to a project-local locations, followed by POM parsing and classpath building. For the rest of what’s needed, technically I can auto-generate a Main.java and a Manifest file, and run various shell commands to produce the uberjar. However we’re not quite there yet, and it would require an expansion of features on my end to support the required configuration for ABCL projects.

The Future

Here are a few potential paths the future could take.

Status Quo: just run ABCL

Tell ABCL users to bundle a version of ABCL as a dependency of their production app, where their “executable” becomes a wrapper around a call to abcl on their code.

Pro: Nothing to do.

Con: Haven’t advanced the state of the art.

Implement uiop/image:dump-image

My personal bias is to not overrely on ASDF, especially where compilers have their own first-class solution. That first-class solution will simply always be better supported than asdf:make, etc. It’s fine to use ASDF to load systems, but beyond that I don’t think it’s its responsibility to handle the production of executables.

Fix asdf-jar + rely on external tooling

Perhaps it’s just a matter of making the Java wrapper shown above more consistent, at which point vend or other tools can use all the existing components to cobble together a runnable fatjar.

Pro: Potentially not too much work.

Con: No push-button solution from ABCL itself.

New core functionality / new contrib

Perhaps in tandem with an expansion to abcl-asdf (or a rewrite), ABCL can provide some ext:fatjar function that:

• Compiles all FASLs and loads them into a JAR.
• Includes all specified Java deps from a customizable location (default to ~/.m2/).
• Includes the ABCL jar itself (probably contrib too).
• Accepts a :main or :entry keyword arg which accepts an entrypoint symbol.
• Produces a Main.class that internally invokes the entrypoint (similar to what’s shown above).

So the function could look something like:

    (ext:fatjar :my-project
                :main #'my-project:launch
                :java #p"/home/me/code/my-project/java-deps/")

Thank you for taking the time to read and consider this. Please let me know your thoughts.

[easye]

Thanks for the summary of what exists, what works, what should exist.

One point of clarification: in your anticipated scenario, can the JVM consuming the fatjar have a local filesystem to write to? Unpacking artifacts from a jar onto a local filesystem and then arranging for them to be accessible from a minimal ABCL runtime should be considerably easier than figuring out how to load everything from the JAR itself.

An additional point of inquiry would be: how important is startup speed?

[easye]

• If you haven’t already manually loaded abcl-contrib and abcl-asdf before loading this system, ASDF doesn’t know what to do with the :mvn blocks and throws an error.

One can get ASDF to throw more sensible errors by including an :defsystem-depends-on clause like https://github.com/easye/jeannie/blob/master/jeannie.asd#L7, and may create additional ASDF "shims" that overload asdf:load-op to require ABCL-CONTRIB.

[fosskers]

Thanks for reading through all of that.

can the JVM consuming the fatjar have a local filesystem to write to?

I don't see why not. Would said "minimal ABCL runtime" also be inside that fatfar?

how important is startup speed?

Not at all, in my mind. I think Java people are used to slow JVM startup times. Are you hinting at compiling the fasls then and there on the host machine, as opposed to bundling them ahead of time? My only stipulation on "provisioning" would be that all of this would have to work without a network connection, namely, I should be able to run one of these fatjars on a plane.

Similarly, I consider Graal out of scope here (as a means to improve startup time). One step at a time 😆

[easye]

Are you hinting at compiling the fasls then and there on the host machine, as opposed to bundling them ahead of time?

That would certainly be one shortcut made possible by assuming locally writable storage. But the more important implementation strategy enabled by this requirement is that one doesn't have to ensure everything has to be loaded directly from the jar, but can have the bootstrap routine copy the necessary bits and bytes to the filesystem, set the necessary paths, and then take it from there.

[mtstickney]

I don't have a lot of concrete detail here (I'm still fooling with this myself), and it doesn't address the maven/fatjar portion of this as much, but this is something I was leaning towards trying as a jar-oriented packaging method:

1. Build a jar with the fasls (also the source if you like, but remember, not everybody wants to ship their source code with an app)
2. Include a system definition that understands precompiled files.[1]
3. Add the path to the (lisp) system in the jar to the ASDF registry using ABCL's support for jar-based pathnames[2], and then load the system (seems like it would require a non-lisp stub, but maybe not).

[1] This might be either a precompiled-system system, or a regular system that uses compiled-file for each of the file components. I haven't used these before, but it looks like precompiled-system is meant to load a system that exists as a single fasl file, e.g. one that was built with compile-bundle-op or monolithic-compile-bundle-op, which is certainly an option but not exactly typical.

[2] I haven't checked yet, but it seems likely that you could get the jar path out of the classloader during startup, so you wouldn't have a problem with absolute pathnames.

In theory, ASDF should be able to load the FASLs directly out of the jar without needing addition extraction, and without compiling source or trying to trick it into thinking some source had already been compiled (i.e. by fooling with output-translations). That would depend on ABCL implementing enough operations on jar-based pathnames for ASDF to do whatever it does during a load, though, and I'm not sure if that's the case. If it weren't, extracting the fasls to a temporary directory would probably do the trick, although that's going to limit the valid use-cases and impose a performance penalty.