outlines llamacpp_example is broken

Describe the issue as clearly as possible:

examples/llamcpp_example.py is broken

It seems like the model is producing some garbage output (which shouldnt be allowed by the logit processor). Posted my debug below.

Steps/code to reproduce the bug:

from enum import Enum

from pydantic import BaseModel, constr

import outlines
import llama_cpp


class Weapon(str, Enum):
    sword = "sword"
    axe = "axe"
    mace = "mace"
    spear = "spear"
    bow = "bow"
    crossbow = "crossbow"


class Armor(str, Enum):
    leather = "leather"
    chainmail = "chainmail"
    plate = "plate"


class Character(BaseModel):
    name: constr(max_length=10)
    age: int
    armor: Armor
    weapon: Weapon
    strength: int


if __name__ == "__main__":
    # curl -L -o mistral-7b-instruct-v0.2.Q5_K_M.gguf https://huggingface.co/TheBloke/Mistral-7B-Instruct-v0.2-GGUF/resolve/main/mistral-7b-instruct-v0.2.Q5_K_M.gguf
    model = outlines.models.LlamaCpp(
        model=llama_cpp.Llama(model_path="./mistral-7b-instruct-v0.2.Q5_K_M.gguf")
    )

    # Construct structured sequence generator
    generator = outlines.generate.json(model, Character)

    # Draw a sample
    seed = 789005

    prompt = "Instruct: You are a leading role play gamer. You have seen thousands of different characters and their attributes.\nPlease return a JSON object with common attributes of an RPG character. Give me a character description\nOutput:"

    sequence = generator(prompt, seed=seed, max_tokens=512)
    print(sequence)

Expected result:

Valid Character Type

Error message:

Traceback (most recent call last):
  File "/Users/bharathc/Projects/temp.py", line 48, in <module>
  File "/Users/bharathc/Projects/.env/lib/python3.12/site-packages/outlines/generate/ap
i.py", line 512, in __call__
    return self._format(completions)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/bharathc/Projects/.env/lib/python3.12/site-packages/outlines/generate/ap
i.py", line 488, in _format
    return self.format_sequence(sequences)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/bharathc/Projects/.env/lib/python3.12/site-packages/outlines/generate/js
on.py", line 50, in <lambda>
    generator.format_sequence = lambda x: schema_object.parse_raw(x)
                                          ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/bharathc/Projects/.env/lib/python3.12/site-packages/pydantic/main.py", l
ine 1214, in parse_raw
    raise pydantic_core.ValidationError.from_exception_data(cls.__name__, [error])
pydantic_core._pydantic_core.ValidationError: 1 validation error for Character
__root__
  Expecting value: line 1 column 1 (char 0) [type=value_error.jsondecode, input_value='Abstract ERROR
 ERRORAbst...bstractAbstractAbstract', input_type=str]

Outlines/Python version information:

Outlines version 0.1.1

Python version Python 3.12.5 (main, Aug 6 2024, 19:08:49) [Clang 15.0.0 (clang-1500.1.0.2.5)]

Context for the issue:

I put a breakpoint here

and it seems the value of completions is

(Pdb) completions
'Abstract ERROR ERRORAbstract ERROR ERRORAbstract ERRORAbstract ERRORAbstract ERRORAbstract ERRORAbst
ract ERRORAbstractAbstract ERRORAbstractAbstract ERRORAbstractAbstractAbstract ERRORAbstractAbstractA
bstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstra
ctAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbs
tractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstract
AbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstr
actAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAb
stractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstrac
tAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbst
ractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractA
bstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstra
ctAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbs
tractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstract
AbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstr
actAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAb
stractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstrac
tAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbst
ractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractA
bstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstra
ctAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbs
tractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstract
AbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstr
actAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAb
stractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstrac
tAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbst
ractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractA
bstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstra
ctAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbs
tractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstract
AbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstr
actAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAb
stractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstrac
tAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbst
ractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractA
bstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstra
ctAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbs
tractAbstractAbstractAbstractAbstractAbstractAbstractAbstract'

So not sure whats going on, I guess llama cpp is not using the logit processor correctly?

Nov 08 '24 04:11 bharathc346

I'm having a similar experience trying the named entity extraction example from the cookbook using llama_cpp with a local model (zephyr-7b) on an M1 Mac.

Here is the interesting part of the error: ValidationError: 1 validation error for Order __root__ Expecting value: line 1 column 1 (char 0) [type=value_error.jsondecode, input_value='AbstractAbstractAbstract...bstractAbstractAbstract', input_type=str]

Looks like outlines is making the model output lots of 'Abstract'.

Outlines version 0.1.3. Python version 3.11.7 llama_cpp_python version 0.3.1

Nov 12 '24 10:11 axel-at-drom

I'm curious, @bharathc346, why your code has this:

model = outlines.models.LlamaCpp(
    model=llama_cpp.Llama(model_path="./mistral-7b-instruct-v0.2.Q5_K_M.gguf")
)

Whereas the code I find in the repository has this:

model = outlines.models.llamacpp("./mistral-7b-instruct-v0.2.Q5_K_M.gguf")

But clearly the code in the repo cannot be correct because outlines.models.llamacpp doesn't take a local filename. If you had to change the example just to get it to run, that would be another way the example is broken.

Are these examples not actually being tested, or what?

Nov 28 '24 09:11 furrykef

Are these examples not actually being tested, or what?

Feel free to update the documentation so the examples are tested in CI, that would be a great contribution.

Nov 28 '24 20:11 rlouf