clj-nix

Nix helpers for Clojure projects

STATUS: alpha. Please leave feedback.

Table of contents

• Introduction
• Usage
  • Generate lock file
  • API
  • GitHub action
• FAQ
• Tutorial
• Similar projects

Introduction

The main goal of the project is to reduce the friction between Clojure and Nix. Nix is a great tool to build and deploy software, but Clojure is not well supported in the Nix ecosystem.

clj-nix tries to improve the situation, providing Nix helpers to interact with Clojure projects

The main difficulty of packaging a Clojure application with Nix is that the derivation is restricted from performing any network request. But Clojure does network requests to resolve the dependency tree. Some network requests are done by Maven, since Clojure uses Maven under the hood. On the other hand, since git deps were introduced, Clojure also access the network to resolve the git dependencies.

A common solution to this problem are lock files. A lock file is a snapshot of the entire dependency tree, usually generated the first time we install the dependencies. Subsequent installations will use the lock file to install exactly the same version of every dependency. Knowing beforehand all the dependencies, we can download and cache all of them, avoiding network requests during the build phase with Nix.

Ideally, we could reuse a lock file generated by Maven or Clojure itself, but lock files are not popular in the JVM/Maven ecosystem. For that reason, clj-nix provides a way to create a lock file from a deps.edn file. Creating a lock file is a prerequisite to use the Nix helpers provided by clj-nix

GOALS:

• Create a binary from a clojure application
• Create an optimized JDK runtime to execute the clojure binary
• Create GraalVM native images from a clojure application
• Simplify container creation for a Clojure application
• Run any arbitrary clojure command at Nix build time (like clj -T:build or clj -M:test)

Usage

This project requires Nix Flakes

New project template

nix flake new --template github:jlesquembre/clj-nix ./my-new-project
cd ./my-new-project
git init
git add .

Remember that with flakes, only the files tracked by git are recognized by Nix.

Templates are for new projects. If you want to add clj-nix to an existing project, I suggest just copy the parts you need from the template (located here: clj-nix/templates/default)

Generate lock file

As mentioned, a lock file must be generated in advance:

nix run github:jlesquembre/clj-nix#deps-lock

That command looks for deps.edn files in your project and generates a deps-lock.json file in the current directory. Remember to re-run it if you update your dependencies.

It is possible to add the dependencies to the nix store during the lock file generation. Internally we are invoking the nix store add-path command. By default, it's disabled because that command is relatively slow. To add the dependencies to the nix store, set the environment variable CLJNIX_ADD_NIX_STORE to true, e.g.:

CLJNIX_ADD_NIX_STORE=true nix run github:jlesquembre/clj-nix#deps-lock

Ignore deps.edn files

Sometimes it could be useful to ignore some deps.edn files, to do that, just pass the list of files to ignore the the deps-lock command:

nix run github:jlesquembre/clj-nix#deps-lock -- ignore/deps.edn

Leiningen

Leiningen projects are supported. Use the --lein option to add the project.clj dependencies to the lock file. This option can be combined with ignored files:

nix run github:jlesquembre/clj-nix#deps-lock -- --lein ignore/deps.edn

Keep in mind that deps-lock command is not optimized for Leiningen projects, it will download all the maven dependencies every time we generate the lock file. For that reason, it is recommended to add a deps.edn file with the same dependencies to Leiningen projects. That way, we reduce the number of network requests when the deps-lock command is invoked.

There are projects to automatically generate a deps.edn file from a Leiningen project (e.g.: depify)

API

Derivations:

• mkCljBin: Creates a clojure application
• customJdk: Creates a custom JDK with jlink. Optionally takes a derivation created with mkCljBin. The intended use case is to create a minimal JDK you can deploy in a container (e.g: a Docker image)
• mkGraalBin: Creates a binary with GraalVM from a derivation created with mkCljBin
• mkCljLib: Creates a clojure library jar
• mkBabashka: Builds custom babashka

NOTE: Extra unknown attributes are passed to the mkDerivation function, see mkCljBin section for an example about how to add a custom check phase.

Helpers:

• mkCljCli: Takes a derivation created with customJdk and returns a valid command to launch the application, as a string. Useful when creating a container.
• bbTasksFromFile: Helper to wrap all the clojure functions in a file as bash scripts. Useful to create a nix develpment shell with devshell
• mk-deps-cache: Creates a Clojure deps cache (maven cache + gitlibs cache). Used by mkCljBin and mkCljLib. You can use this function to to have access to the cache in a nix derivation.

mkCljBin

Creates a Clojure application. Takes the following attributes (those without a default are mandatory, extra attributes are passed to mkDerivation):

• jdkRunner: JDK used at runtime by the application. (Default: jdk)

• projectSrc: Project source code.

• name: Derivation and clojure project name. It's recommended to use a namespaced name. If not, a namespace is added automatically. E.g. foo will be transformed to foo/foo

• version: Derivation and clojure project version. (Default: DEV)

• main-ns: Main clojure namespace. A -main function is expected here.

• buildCommand: Command to build the jar application. If not provided, a default builder is used: build.clj. If you provide your own build command, clj-nix expects that a jar will be generated in a directory called target

• lockfile: The lock file. (Default: ${projectSrc}/deps-lock.json)

Example:

mkCljBin {
  jdkRunner = pkgs.jdk17_headless;
  projectSrc = ./.;
  name = "me.lafuente/clj-tuto";
  version = "1.0";
  main-ns = "demo.core";

  buildCommand = "clj -T:build uber";

  # mkDerivation attributes
  doCheck = true;
  checkPhase = "clj -M:test";
}

Outputs:

• out: The application binary
• lib: The application jar

customJdk

Creates a custom JDK runtime. Takes the following attributes (those without a default are mandatory):

• jdkBase: JDK used to build the custom JDK with jlink. (Default: nixpkgs.jdk17_headless)

• cljDrv: Derivation generated with mkCljBin.

• name: Derivation name. (Default: cljDrv.name)

• version: Derivation version. (Default: cljDrv.version)

• jdkModules: Option passed to jlink --add-modules. If null, jeps will be used to analyze the cljDrv and pick the necessary modules automatically. (Default: null)

• multiRelease: Option passed to jdeps --multi-release. Should be an integer >=9 or a boolean. If true, the value is set to base. If false or not specified, clj-nix will try to detect if the jar is a multi-release jar and set the value automatically. See The jdeps Command for more info. (Default: false)

• locales: Option passed to jlink --include-locales. (Default: null)

Example:

customJdk {
  jdkBase = pkgs.jdk17_headless;
  name = "myApp";
  version = "1.0.0";
  cljDrv = myCljBinDerivation;
  locales = "en,es";
}

Outputs:

• out: The application binary, using the custom JDK
• jdk: The custom JDK

mkGraalBin

Generates a binary with GraalVM from an application created with mkCljBin. Takes the following attributes (those without a default are mandatory):

• cljDrv: Derivation generated with mkCljBin.

• graalvm: GraalVM used at build time. (Default: nixpkgs.graalvmCEPackages.graalvm17-ce)

• name: Derivation name. (Default: cljDrv.name)

• version: Derivation version. (Default: cljDrv.version)

• extraNativeImageBuildArgs: Extra arguments to be passed to the native-image command. (Default: [ ])

• graalvmXmx: XMX size of GraalVM during build (Default: "-J-Xmx6g")

Example:

mkGraalBin {
  cljDrv = myCljBinDerivation;
}

An extra attribute is present in the derivation, agentlib, which generates a script to help with the generation of a reflection config file

mkCljLib

Creates a jar file for a Clojure library. Takes the following attributes (those without a default are mandatory, extra attributes are passed to mkDerivation):

• projectSrc: Project source code.

• name: Derivation and clojure library name. It's recommended to use a namespaced name. If not, a namespace is added automatically. E.g. foo will be transformed to foo/foo

• version: Derivation and clojure project version. (Default: DEV)

• buildCommand: Command to build the jar application. If not provided, a default builder is used: jar fn in build.clj. If you provide your own build command, clj-nix expects that a jar will be generated in a directory called target

Example:

mkCljLib {
  projectSrc = ./.;
  name = "me.lafuente/my-lib";
  buildCommand = "clj -T:build jar";
}

mkBabashka

Builds Babashka with the specified features. See babashka feature flags for the full list. Notice that the feature names in the Nix wrapper are case insensitive and we can omit the BABASHKA_FEATURE_ prefix.

Takes the following attributes:

• withFeatures: List of extra Babashka features. (Default: [])

• bbLean: Disable default Babashka features. (Default: false)

• graalvm: GraalVM used at build time. (Default: nixpkgs.graalvmCEPackages.graalvm11-ce)

Example:

mkBabashka {
  withFeatures = [ "jdbc" "sqlite" ];
}

mkCljCli

Returns a string with the command to launch an application created with customJdk. Takes the following attributes (those without a default are mandatory):

jdkDrv: Derivation generated with customJdk

java-opts: Extra arguments for the Java command (Default: [])

extra-args: Extra arguments for the Clojure application (Default: "")

Example:

mkCljCli {
  jdkDrv = self.packages."${system}".jdk-tuto;
  java-opts = [ "-Dclojure.compiler.direct-linking=true" ];
  extra-args = [ "--foo bar" ];
}

bbTasksFromFile

Reads a Clojure file, for each function generates a devshell command. Takes a path (to a Clojure file) or the following attributes:

file: Path to the Clojure file

bb: Babashka derivation (Default: nixpkgs.babashka)

Example:

devShells.default =
  pkgs.devshell.mkShell {
    commands = pkgs.bbTasksFromFile ./tasks.clj;
    # or
    commands = pkgs.bbTasksFromFile {
      file = ./tasks.clj;
      bb = pkgs.mkBabashka { withFeatures = [ "jdbc" "sqlite" ]; };
    };
  }

mk-deps-cache

Generate maven + gitlib cache from a lock file. This is a lower level helper, usually you want to use mkCljBin or mkCljLib and define a custom build command with the buildCommand argument.

lockfile: deps-lock.json file

Example:

mk-deps-cache {
  lockfile = ./deps-lock.json;
}

Github action

It's possible to add a GitHub action to automatically update the deps-lock.json file on changes:

name: "Update deps-lock.json"
on:
  push:
    paths:
      - "**/deps.edn"

jobs:
  update-lock:
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v3

      - uses: cachix/install-nix-action@v17

      - name: Update deps-lock
        run: "nix run github:jlesquembre/clj-nix#deps-lock"

      - name: Create Pull Request
        uses: peter-evans/[email protected]
        with:
          commit-message: Update deps-lock.json
          title: Update deps-lock.json
          branch: update-deps-lock

FAQ

• How can I define extra runtime dependencies?

If you try to call a external program from your Clojure application, you will get an error (something like Cannot run program "some-program": error=2, No such file or directory). One possible solution is to wrap the binary:

cljpkgs.mkCljBin {
# ...
nativeBuildInputs = [ pkgs.makeWrapper ];
postInstall = ''
  wrapProgram $cljBinary \
    --set PATH ${pkgs.lib.makeBinPath [ pkgs.cowsay ]}
'';
}

Notice that the $cljBinary is a proper Bash variable. It is created by mkCljBin during the install phase.

or if you want to define the dependencies in a docker image:

pkgs.dockerTools.buildLayeredImage {
  # ...
  config = {
    Env = [ "PATH=${pkgs.lib.makeBinPath [ pkgs.cowsay ]}" ];
  };
};

Tutorial

Source code for this tutorial can be found here: https://github.com/jlesquembre/clj-demo-project

Init

There is a template to help you start your new project:

nix flake new --template github:jlesquembre/clj-nix ./my-new-project

For this tutorial you can clone the final version:

git clone [email protected]:jlesquembre/clj-demo-project.git

First thing we need to do is to generate a lock file:

nix run github:jlesquembre/clj-nix#deps-lock
git add deps-lock.json

NOTE: The following examples assume that you cloned the demo repository, and you are executing the commands from the root of the repository. But with Nix flakes, it's possible to point to the remote git repository. E.g.: We can replace nix run .#foo with nix run github:/jlesquembre/clj-demo-project#foo

Create a binary from a Clojure application

First, we create a new package in our flake:

clj-tuto = cljpkgs.mkCljBin {
  projectSrc = ./.;
  name = "me.lafuente/cljdemo";
  main-ns = "demo.core";
};

Let's try it:

nix build .#clj-tuto
./result/bin/clj-tuto
# Or
nix run .#clj-tuto

Nice! We have a binary for our application. But how big is our app? We can find it with:

nix path-info -sSh .#clj-tuto
# Or to see all the dependencies:
nix path-info -rsSh .#clj-tuto

Um, the size of our application is 1.3G, not ideal if we want to create a container. We can use a headless JDK to reduce the size, let's try that:

clj-tuto = cljpkgs.mkCljBin {
  projectSrc = ./.;
  name = "me.lafuente/cljdemo";
  main-ns = "demo.core";
  jdkRunner = pkgs.jdk17_headless;
};

nix build .#clj-tuto
nix path-info -sSh .#clj-tuto

Good, now the size is 703.9M. It's an improvement, but still big. To reduce the size, we can use the customJdk helper.

Create custom JDK for a Clojure application

We add a package to our flake, to build a customized JDK for our Clojure application:

jdk-tuto = cljpkgs.customJdk {
  cljDrv = self.packages."${system}".clj-tuto;
  locales = "en,es";
};

nix build .#jdk-tuto
nix path-info -sSh .#jdk-tuto

Not bad! We reduced the size to 96.3M. That's something we can put in a container. Let's create a container with our application.

Create a container

Again, we add a new package to our flake, in this case it will create a container:

clj-container =
  pkgs.dockerTools.buildLayeredImage {
    name = "clj-nix";
    tag = "latest";
    config = {
      Cmd = clj-nix.lib.mkCljCli self.packages."${system}".jdk-tuto { };
    };
  };

nix build .#clj-container
nix path-info -sSh .#clj-container

The container's size is 52.8M. Wait, how can be smaller than our custom JDK derivation? There are 2 things to consider.

First, notice that we used the mkCljCli helper function. In the original version, our binary is a bash script, so bash is a dependency. But in a container we don't need bash, the container runtime can launch the command, and we can reduce the size by removing bash

Second, notice that the image was compressed with gzip.

Let's load and execute the image:

docker load < result
docker run -it --rm clj-nix
docker images

Docker reports an image size of 99.2MB

Create a native image with GraalVM

If we want to continue reducing the size of our derivation, we can compile the application with GraalVM. Keep in mind that size it's not the only factor to consider. There is a nice slide from the GraalVM team, illustrating what technology to use for which use case:

(The image was taken from a tweet by Thomas Würthinger)

For more details, see: Does GraalVM native image increase overall application performance or just reduce startup times?

Let's compile our Clojure application with GraalVM:

graal-tuto = cljpkgs.mkGraalBin {
  cljDrv = self.packages."${system}".clj-tuto;
};

nix build .#graal-tuto
./result/bin/clj-tuto
nix path-info -sSh .#graal-tuto

The size is just 43.4M.

We can create a container from this derivation too:

graal-container =
  let
    graalDrv = self.packages."${system}".graal-tuto;
  in
  pkgs.dockerTools.buildLayeredImage {
    name = "clj-graal-nix";
    tag = "latest";
    config = {
      Cmd = "${graalDrv}/bin/${graalDrv.name}";
    };
  };

docker load < result
docker run -it --rm clj-graal-nix

In this case, the container image size is 45.3MB, aproximately half the size of the custom JDK image.

Similar projects

• dwn
• clj2nix
• mvn2nix
• clojure-nix-locker