Avoid top-level imports in `cli.py`

[AdrianVollmer]

Outputing the help takes about one second on my system:

$ time llm --help > /dev/null
llm --help > /dev/null  0.82s user 0.10s system 99% cpu 0.920 total

That's because cli.py is importing a lot of libraries at the top-level. I propose moving the imports in that file to the functions where they are needed. It may not look asthetic, but it will speed up the call to --help by many orders of magnitudes, and probably somewhat speed up the warm up time when calling subcommands.

Let me know if I can help out with a PR. (It's probably also something that an LLM would be good at.)

Edit: I'm seeing that __init__.py is also affected, and my suggestions probably doesn't apply there ...

[simonw]

I'd be interested in a benchmark that shows how much time we could save here.

[AdrianVollmer]

What a coincidence, I went ahead and implemented it and just pushed my proposed changes. I had to split up the package into two, though. Still need to run the test suite. See the commit message for a basic benchmark. I can create a benchmark for the other subcommands, but I expect the difference to be mostly negligible.

[AdrianVollmer]

Yeah, unfortunately this won't work. The plugin mechanism is too intertwined with the CLI.

[rpdelaney]

I'd be interested in a benchmark that shows how much time we could save here.

Based on OP's assertion in #922 that LLM_LOAD_PLUGINS has no effect, I have a branch up in my fork that adds an override for DEFAULT_PLUGINS with an env var:

diff to llm/plugins.py

diff --git c/llm/plugins.py w/llm/plugins.py
index 5c00b9e..46b30c8 100644
--- c/llm/plugins.py
+++ w/llm/plugins.py
@@ -5,7 +5,7 @@ import pluggy
 import sys
 from . import hookspecs
 
-DEFAULT_PLUGINS = ("llm.default_plugins.openai_models",)
+DEFAULT_PLUGINS: tuple[str] = (os.getenv("LLM_DEFAULT_PLUGINS", "llm.default_plugins.openai_models"),)
 
 pm = pluggy.PluginManager("llm")
 pm.add_hookspecs(hookspecs)
@@ -43,5 +43,6 @@ def load_plugins():
                 sys.stderr.write(f"Plugin {package_name} could not be found\n")
 
     for plugin in DEFAULT_PLUGINS:
-        mod = importlib.import_module(plugin)
-        pm.register(mod, plugin)
+        if len(plugin):
+            mod = importlib.import_module(plugin)
+            pm.register(mod, plugin)

Here's what I'm seeing:

Control group

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
llm --help	522.5 ± 4.8	517.0	530.9	1.00
llm models	522.1 ± 4.7	515.3	529.4	1.00
llm plugins	522.3 ± 6.4	515.4	535.8	1.00
llm templates	557.3 ± 13.3	542.7	586.9	1.00

Test group (LLM_DEFAULT_PLUGINS="")

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
llm --help	241.1 ± 1.3	238.3	242.9	1.00
llm models	240.7 ± 2.1	238.1	244.2	1.00
llm plugins	245.4 ± 5.5	238.5	258.3	1.00
llm templates	256.3 ± 8.6	247.1	274.6	1.00

I use the llm-ollama plugin for local execution, and I saw too big a deviation to trust those results. If anyone would like to generate similar benchmarks with a remote-API plugin, here's the script I used to generate these benchmarks with hyperfine.

#!/usr/bin/env bash
#

set -o nounset     # expanding unset variables is fatal

if ! command -v hyperfine >/dev/null 2>&1 ; then echo "Missing dependency: hyperfine" 1>&2 ; exit 1 ; fi
if ! command -v sed >/dev/null 2>&1 ; then echo "Missing dependency: sed" 1>&2 ; exit 1 ; fi
if ! command -v llm >/dev/null 2>&1 ; then echo "Missing dependency: llm" 1>&2 ; exit 1 ; fi

testreport="test.md"
controlreport="control.md"

fine() {
  : # Run a benchmark on a command with hyperfine
  test_command="$1"
  report_file="$2"
  tmpfile="$(mktemp)"

  hyperfine --export-markdown "$tmpfile" -- "$test_command" || exit

  if [[ -e "$report_file" ]] ; then
    # remove the table headers (so we don't repeat them)
    sed -i '1,2d' "$tmpfile"
  fi
  # append this run to the report
  cat "$tmpfile" >> "$report_file"
  rm -f "$tmpfile"
}

cmdbench() {
  : # Compare performance of a command with LLM_DEFAULT_PLUGINS set, and unset
  test_command="$1"
  (fine "$test_command" "$controlreport")
  (export LLM_DEFAULT_PLUGINS=""; fine "$test_command" "$testreport")
}

main() {
  cmdbench "llm --help"
  cmdbench "llm models"
  cmdbench "llm plugins"
  cmdbench "llm templates"
}

if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then
  main "$@"
fi

# EOF

[simonw]

I actually made a bunch of changes to popular plugins based around this same problem that imports could be expensive (especially chunky imports like PyTorch):

• #949

[simonw]

The big challenge here is that other plugins can provide extra commands, which means llm --help HAS to execute plugin loads in order to correctly show the available commands.

[rpdelaney]

I suppose you'd have to maintain a registry of added commands that gets updated when each plugin is installed.

Related to #992, making all plugins optional would be useful for people like me, who don't use OpenAI.