Is AzureOpenAI, Chat Completion, and Tools broken

[arthurgreef]

The bug I get the following error when running the code below. Is it just not possible to use AOAI with chat completion and tools anymore? Thanks.

The model attempted to generate b"Thought 1: I should type the letter 'd'...." after the prompt b"...: 'dog'<|im_end|>\n<|im_start|>assistant\n", but that does not match the given grammar constraints! Since your model is a remote API that does not support full guidance integration we cannot force the model to follow the grammar, only flag an error when it fails to match. You can try to address this by improving the prompt, making your grammar more flexible, rerunning with a non-zero temperature, or using a model that supports full guidance grammar constraints.

To Reproduce Give a full working code snippet that can be pasted into a notebook cell or python file. Make sure to include the LLM load step so we know which model you are using.

""" Type letter evaluation test. """

import os
import time
from guidance import assistant, gen, guidance, models, system, user


def env_or_fail(var_name: str) -> str:
    """Get an environment variable or fail with a message."""
    env_value = os.getenv(var_name, None)
    assert env_value is not None, f"Env '{var_name}' not found"
    return env_value


azure_ai_endpoint = env_or_fail("AZURE_AI_CHAT_ENDPOINT")
azure_ai_key = env_or_fail("AZURE_AI_CHAT_KEY")
model = env_or_fail("AZURE_AI_CHAT_MODEL")

lm = models.AzureOpenAI(
    model=model,
    azure_endpoint=azure_ai_endpoint,
    api_key=azure_ai_key,
    echo=False,
    max_streaming_tokens=2000,
)
assert isinstance(lm, models.AzureOpenAI)

@guidance()
def type_letter(lm, letter):
    """Type a letter."""
    lm += letter
    return lm


tools = {"type_letter": "Returns a letter after it is typed."}
tool_map = {"type_letter": type_letter}

PROMPT = """Type the user message one letter at a time. You have access only to the following tools:

{tools}

Use the following format:

Message: the user message you must type
Thought 1: you should always think about what to do
Action 1: the action to take, has to be one of {tool_names}
Observation 1: the result of the action
... (this Thought/Action/Action Input/Observation can repeat N times)
Thought N: I have typed each letter of the message.
Final Thought: the list of letters that were typed.
Done.

Example:
Message: cat
Thought 1: I should type the letter 'c'.
Action 1: type_letter('c')
Observation 1: 'c'
Thought 2: I should type the letter 'a'.
Action 2: type_letter('a')
Observation 2: 'a'
Thought 3: I should type the letter 't'.
Action 3: type_letter('t')
Observation 3: 't'
Thought 4: I have typed each letter of the message.
Final Thought: I typed the letters: 'c','a','t'
Done.
"""

I = 0
MAX = 1

while I < MAX:

    lm.engine.reset_metrics()

    with system():
        tool_names = list(tools.keys())
        lm += PROMPT.format(tools=tools, tool_names=tool_names)

    with user():
        # lm += "'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ'"
        lm += "Message: 'dog'"

    with assistant():
        lm += gen("trace", max_tokens=2000, tools=[type_letter])

    print(str(lm))
    # print(lm["trace"])
    FINAL_THOUGHT = None
    for line in lm["trace"].split("\n"):
        if line.startswith("Final Thought"):
            FINAL_THOUGHT = line
            break
    print(FINAL_THOUGHT)
    # print(f"{lm.engine.metrics=}")

    I += 1
    time.sleep(60)

System info (please complete the following information):

• OS Windows:
• Guidance Version (guidance.__version__): 0.1.15