feature: nix based scripts

[DrRuhe]

Overview

Enable users to define scripts using Nix, leveraging its package management to ensure reproducible environments, precise dependency management, and access to Nixpkgs.

Benefits of Nix Integration

Central Nixpkgs Revision

Windmill can use a shared Nixpkgs revision, ensuring consistent versions of tools across all scripts. This centralization makes updating and managing dependencies easier, as only the nixpkgs reference has to be updated for all scripts to receive new versions.

Per-Script Dependency Isolation

Nix allows each script to declare its specific environment, avoiding global dependencies. For example, if only one script requires a particular Python library, it can install and use it independently, preventing conflicts with other scripts. This ensures robustness and avoids breaking other scripts due to version mismatches.

Efficient Distribution via the Nix Store

Once a derivation (Nix-built package) is created, it’s stored in the Nix store. Scripts can reuse these pre-built binaries, ensuring fast deployment and execution without recompilation.

Questions

• How to get jsonschema from the tools?
  • nix doesn't automatically produce jsonschema for derivations. For nix derivations that produce scripts in languages already supported by windmill, the existing infrastructure for jsonschemas can be used.
  • in the remaining cases a jsonschema could be added manually in the nix derivation under the meta attribute
  • Is it possible to omit jsonschema?

Example 1 : build a Go tool

An example script using Nix to build a Go tool:

Note that this example is the definition of the carapace tool from nixpkgs.

{ lib, buildGoModule, fetchFromGitHub }:

buildGoModule rec {
  pname = "carapace";
  version = "1.0.7";

  src = fetchFromGitHub {
    owner = "carapace-sh";
    repo = "carapace-bin";
    rev = "v${version}";
    hash = "sha256-...";
  };

  ldflags = [
    "-s" "-w" "-X main.version=${version}"
  ];

  meta = with lib; {
    description = "Multi-shell argument completer";
    license = licenses.mit;
  };
}

Example 2: A python script

This script takes an input MP4 file, extracts the audio using ImageMagick, and generates a spectrogram using Python libraries.

Note that this is a contrived example and the script was generated by ChatGPT, so it likely wont work as is. But that's beside the point of this issue. I believe it still conveys the desire for nix based script support in windmill

{ lib, python3, imagemagick, ffmpeg, matplotlib, numpy, scipy }:

# Define a Python environment with the required tools and libraries
python3.buildPythonApplication rec {
  pname = "audio-spectrogram";
  version = "1.0";

  # Inline Python script
  src = ''
    import numpy as np
    import matplotlib.pyplot as plt
    import subprocess
    from scipy.io import wavfile

    # Step 1: Extract audio from MP4 using ImageMagick
    def extract_audio(mp4_file, output_wav):
        subprocess.run(['convert', mp4_file, output_wav])

    # Step 2: Generate and plot the spectrogram
    def plot_spectrogram(wav_file, output_image):
        sample_rate, audio_data = wavfile.read(wav_file)
        plt.specgram(audio_data, Fs=sample_rate)
        plt.title('Spectrogram')
        plt.savefig(output_image)

    if __name__ == "__main__":
        mp4_input = "input.mp4"
        wav_output = "output.wav"
        spectrogram_output = "spectrogram.png"
        
        extract_audio(mp4_input, wav_output)
        plot_spectrogram(wav_output, spectrogram_output)
  '';

  propagatedBuildInputs = [
    imagemagick   # Tool for extracting audio from video
    ffmpeg        # Handles multimedia files
    matplotlib    # For plotting the spectrogram
    numpy         # For numerical computations
    scipy         # For signal processing
  ];

  buildInputs = [ imagemagick ];

  meta = with lib; {
    description = "Extract audio from an MP4 and plot its spectrogram";
    license = licenses.mit;
    maintainers = with maintainers; [ "yourname" ];
  };
}