Template with escaped vars fails

[tylerbrandt]

Summary

I was trying to include some JSON schema in my template and I ran across this bug where if the input contains an escaped identifier like $$var (per https://docs.python.org/3/library/string.html#template-strings), llm -t crashes.

Repro

1. Create template with escaped string

llm --system 'Hello $$world' --save interpolation-bug

2. Invoke template (it doesn't matter what's in the prompt)

llm -t interpolation-bug 'oops'
Traceback (most recent call last):
  File "/Users/tyler.brandt/Library/Python/3.11/bin/llm", line 8, in <module>
    sys.exit(cli())
             ^^^^^
  File "/Users/tyler.brandt/Library/Python/3.11/lib/python/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/tyler.brandt/Library/Python/3.11/lib/python/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/Users/tyler.brandt/Library/Python/3.11/lib/python/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/tyler.brandt/Library/Python/3.11/lib/python/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/tyler.brandt/Library/Python/3.11/lib/python/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/tyler.brandt/Library/Python/3.11/lib/python/site-packages/llm/cli.py", line 216, in prompt
    prompt, system = template_obj.evaluate(prompt, params)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/tyler.brandt/Library/Python/3.11/lib/python/site-packages/llm/templates.py", line 31, in evaluate
    system = self.interpolate(self.system, params)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/tyler.brandt/Library/Python/3.11/lib/python/site-packages/llm/templates.py", line 48, in interpolate
    "Missing variables: {}".format(", ".join(missing))
                                   ^^^^^^^^^^^^^^^^^^
TypeError: sequence item 0: expected str instance, NoneType found

It looks like the error arises due to the use of template.pattern in extract_vars which returns a list of a single None value per escaped var; Template.get_identifiers (new in 3.11) does not exhibit this behavior (returns no list items instead for such vars).

[tylerbrandt]

For now I am working around it by using a var $schema and passing the value using -p schema "$(cat file)" but it would be nice to not have to do that.

[ryanhalliday]

Example command for what @tylerbrandt mentioned above:

llm -t your-template -p schema "$(cat your-actual-system-prompt.txt)"

Where your-tempalte.yaml should look something like this:

model: "gpt-4"
system: |
    $schema

Slightly annoying to work around, especially considering $$ escape doesn't work but it's good enough for now.

It does look like either of these PRs would resolve this from a quick look: https://github.com/simonw/llm/pull/469 or https://github.com/simonw/llm/pull/429 @simonw

[rpdelaney]

I got this error although I'm not (intentionally) using escaped vars.

$ llm prompt "reverse a linked list" --template howdoi
Traceback (most recent call last):
  File "/home/ryan/.local/bin/llm", line 8, in <module>
    sys.exit(cli())
             ~~~^^
  File "/home/ryan/.local/pipx/venvs/llm/lib/python3.13/site-packages/click/core.py", line 1161, in __call__
    return self.main(*args, **kwargs)
           ~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/home/ryan/.local/pipx/venvs/llm/lib/python3.13/site-packages/click/core.py", line 1082, in main
    rv = self.invoke(ctx)
  File "/home/ryan/.local/pipx/venvs/llm/lib/python3.13/site-packages/click/core.py", line 1697, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
  File "/home/ryan/.local/pipx/venvs/llm/lib/python3.13/site-packages/click/core.py", line 1443, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ryan/.local/pipx/venvs/llm/lib/python3.13/site-packages/click/core.py", line 788, in invoke
    return __callback(*args, **kwargs)
  File "/home/ryan/.local/pipx/venvs/llm/lib/python3.13/site-packages/llm/cli.py", line 351, in prompt
    prompt, system = template_obj.evaluate(prompt, params)
                     ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^
  File "/home/ryan/.local/pipx/venvs/llm/lib/python3.13/site-packages/llm/templates.py", line 37, in evaluate
    prompt = self.interpolate(self.prompt, params)
  File "/home/ryan/.local/pipx/venvs/llm/lib/python3.13/site-packages/llm/templates.py", line 51, in interpolate
    "Missing variables: {}".format(", ".join(missing))
                                   ~~~~~~~~~^^^^^^^^^
TypeError: sequence item 0: expected str instance, NoneType found
$

~/.config/io.datasette.llm/templates/howdoi.yaml

prompt:
    |
    You are a programmer assistant who knows common algorithms and GNU shell utilities.
    You will output ONLY the code to achieve the user's request.

    For example:

    ---
    User:
        format date in bash
    You:
        ```bash
        date '+%Y-%m-%d'
        ```
    User:
        convert mp4 to a high-quality animated gif
    You:
        ```bash
        infile="$1"
        outfile="$2"
        duration_seconds="${3:3"}

        ffmpeg -i "$infile".mp4 -t "$duration_seconds" -vf "fps=10,scale=320:-1:flags=lanczos,split[s0][s1];[s0]palettegen[p];[s1][p]paletteuse" -loop 0 "$outfile"
        ```
    User:
        create a compressed archive of the current directory
    You:
        ```bash
        tar cvfa pwd.tar.xz .
        ```
    User:
        list all authors in this git tree
    You:
        ```bash
        git log --format='%aN' | sort -u
        ```