Structured outputs with banks

[alex-stoica]

Hi @masci

I saw banks allows for entering the model name e.g. {% completion model="gpt-4o" %} but no direct support for structured outputs. One approach I'd see is to use litellm, from their docs:

messages = [{"role": "user", "content": "List 5 important events in the XIX century"}]

class CalendarEvent(BaseModel):
  name: str
  date: str
  participants: list[str]

class EventsList(BaseModel):
    events: list[CalendarEvent]

resp = completion(
    model="gpt-4o-2024-08-06",
    messages=messages,
    response_format=EventsList
)

using banks I guess this would be changed with

response = completion(
    model="gpt-4o-2024-08-06",
    messages=registry.get(name="xix_events"),
    response_format=EventsList
)

and I guess this would work, but is this the intended use? Is there a recommended way to handle structured outputs directly with banks / wo relying on litellm? I'd imagine something like

response = prompt.text(data = {"some_required_param": "some_value"}, response_format = EventsList) 

[masci]

I'm assuming you're referring to the completion tag:

{% completion model="gpt-4o-2024-08-06" %}
  {% chat role="system" %}You are a helpful assistant{% endchat %}
  {% chat role="user" %}List 5 important events in the XIX century{% endchat %}
{% endcompletion %}

I think something like this should be doable:

{% completion model="gpt-4o-2024-08-06" response_format="events_list" %}
  {% chat role="system" %}You are a helpful assistant{% endchat %}
  {% chat role="user" %}List 5 important events in the XIX century{% endchat %}
{% endcompletion %}

The code using this prompt would be similar to the one you propose, just I would pass more than one format to allow different blocks to have different formats, like this:

response = prompt.text(data = {"some_required_param": "some_value"}, response_formats = {"events_list": EventsList}) 

Would that work?

[alex-stoica]

Tried some code just to see what error would occur

from banks import Prompt
from typing import List
from pydantic import BaseModel, Field


class Recipe(BaseModel): 
    recipe_name: str = Field(
        description="The name of the recipe."
    )
    ingredients: List[str] = Field(
        description="List of ingredients with amounts, e.g., '1 cup of flour'."
    )


prompt_template = """
{% completion model="gemini-2.5-flash" response_format="recipe_list" %}
  {% chat role="system" %}
    You are a helpful culinary assistant
  {% endchat %}
  {% chat role="user" %}
    List a few popular {{ recipe_type }} recipes, and include the amounts of ingredients for each.
  {% endchat %}
{% endcompletion %}
"""

prompt = Prompt(prompt_template)

my_recipes: List[Recipe] = prompt.text(
    data={"recipe_type": "cannoli"},
    response_formats={"recipe_list": List[Recipe]}
) 

and got

jinja2.exceptions.TemplateSyntaxError: Invalid syntax for completion: [Token(lineno=2, type='name', value='model'), Token(lineno=2, type='assign', value='='), Token(lineno=2, type='string', value='gemini-2.5-flash'), Token(lineno=2, type='name', value='response_format'), Token(lineno=2, type='assign', value='='), Token(lineno=2, type='string', value='recipe_list')]

so currently the prompt cannot be constructed as it doesn't know about response_format I guess. One thing might be better in a future implementation is to have a single parameter for structured outputs, not a dict that might have multiple keys. More exactly, to allow for

my_recipes: List[Recipe] = prompt.text(
    data={"recipe_type": "cannoli"},
    response_format=List[Recipe]
) 

instead of what I wrote above. What do you think?

[masci]

so currently the prompt cannot be constructed as it doesn't know about response_format I guess.

Correct, that was pseudo-code just to illustrate a possible user experience.

I like this idea

my_recipes: List[Recipe] = prompt.text(
    data={"recipe_type": "cannoli"},
    response_format=List[Recipe]
) 

Maybe one comment, I would probably introduce a new "verb" in the prompt api, because text() is supposed to always return a string, so something like this instead (all pseudo-code):

prompt_template = """
{% completion model="gemini-2.5-flash" %}
  {% chat role="system" %}
    You are a helpful culinary assistant
  {% endchat %}
  {% chat role="user" %}
    List a few popular {{ recipe_type }} recipes, and include the amounts of ingredients for each.
  {% endchat %}
{% endcompletion %}

Some 
other 
output 
that will be ignored by `prompt.structured()` but will be rendered as usual by `prompt.text()`
"""

my_recipes: List[Recipe] = prompt.structured(
    data={"recipe_type": "cannoli"},
    response_format=List[Recipe]
) 

So the plan would be:

• Introduce a new method structured (or any better name we might find) to the Prompt class
• structured() expects the model use for output validation
• when structured() is called, all the completion blocks will be executed passing response_format=...
• blocks that are not completion will be ignored when structured() is called (this would simplify a lot of things)

WDYT?

[alex-stoica]

Sounds very good, except for the part with

Some 
other 
output 
that will be ignored by `prompt.structured()` but will be rendered as usual by `prompt.text()`

I don't get that - what other content would you envision being here?

I'd add (a little bit unrelated with the subject) that some structured outputs are allowed by gemini and not allowed by gpt5, so this has to be implemented with care. For example

OpenAI requires a pydantic model that has a list of Step objects, but doesn't allow for simply having as structured output just list(Step), whereas Gemini (2.5 flash at least) allows for it. tldlr: structured output doesn't mean exactly the same thing, it depends on the provider, so the implementation should accomodate these variations