Pydantic 1.X changes prompt order and breaks response

[bennnym]

First check

• [X] I added a descriptive title to this issue.
• [X] I used the GitHub search to try to find a similar issue and didn't find one.
• [X] I searched the Marvin documentation for this issue.

Bug summary

When trying to use Marvin with Prefect, prefect requires pydantic < 2.0.

Pydantic seems to order their arguments differently and hence the prompt changes and the response is no longer parsed properly resulting in an erorr.

In the given example the error can be seen below for anything pydantic v1.

Reproduction

# CODE

from marvin import ai_model
import marvin
import enum as Enum
from pydantic import BaseModel, Field

marvin.settings.llm_model = 'openai/gpt-4-0613'

class Speed(Enum):
    """Speed entities"""
    SLOW = "SLOW"
    EVEN = "EVEN"
    FAST = "FAST"
    NONE = "NONE"


@ai_model(temperature=0.01, top_p=0.05)
class WalkingSpeed(BaseModel):
    """
    Comment for how fast someone is walking
    """
    early_speed: Speed = Field(
            ..., description="The speed someone is walking at at the start of their walk."
        )


WalkingSpeed("Slow to begin.") # run me

Using Pydantic V1 in order to use Prefect 2.X

pydantic 1.10.13

The prompt from WalkingSpeed.as_prompt("Slow to begin.")

{
    "messages": [
        {
            "content": "The user will provide text that you need to parse into a\nstructured form .\nTo validate your response, you must call the\n`FormatResponse` function.\nUse the provided text and context to extract, deduce, or infer\nany parameters needed by `FormatResponse`, including any missing\ndata.\n\nYou have been provided the following context to perform your task:\n    - The current time is 2023-09-29 00:37:05.131503+00:00.",
            "role": "system",
        },
        {"content": "The text to parse: Slow to begin.", "role": "user"},
    ],
    "max_tokens": 1500,
    "api_key": 123,
    "model": "gpt-4-0613",
    "temperature": 0.8,
    "request_timeout": 600.0,
    "functions": [
        {
            "parameters": {
                "type": "object",
                "properties": {
                    "early_speed": {
                        "description": "The speed someone is walking at at the start of their walk.",
                        "allOf": [{"$ref": "#/definitions/Speed"}],
                    }
                },
                "required": ["early_speed"],
                "definitions": {
                    "Speed": {
                        "title": "Speed",
                        "description": "Speed entities",
                        "enum": ["SLOW", "EVEN", "FAST", "NONE"],
                    }
                },
            },
            "name": "FormatResponse",
            "description": "Comment for how fast someone is walking",
        }
    ],
    "function_call": {"name": "FormatResponse"},
}

Using Pydantic V2 and pydantic-setttings does not work with Prefect 2.X

pydantic 2.4.2 pydantic-settings 2.0.3

The prompt from WalkingSpeed.as_prompt("Slow to begin.")

{
    "messages": [
        {
            "content": "The user will provide text that you need to parse into a\nstructured form .\nTo validate your response, you must call the\n`FormatResponse` function.\nUse the provided text and context to extract, deduce, or infer\nany parameters needed by `FormatResponse`, including any missing\ndata.\n\nYou have been provided the following context to perform your task:\n    - The current time is 2023-09-29 00:38:07.384485+00:00.",
            "role": "system",
        },
        {"content": "The text to parse: Slow to begin.", "role": "user"},
    ],
    "max_tokens": 1500,
    "api_key": 123,
    "temperature": 0.8,
    "request_timeout": 600.0,
    "model": "gpt-4-0613",
    "functions": [
        {
            "parameters": {
                "$defs": {
                    "Speed": {
                        "description": "Speed entities",
                        "enum": ["SLOW", "EVEN", "FAST", "NONE"],
                        "title": "Speed",
                        "type": "string",
                    }
                },
                "properties": {
                    "early_speed": {
                        "allOf": [{"$ref": "#/$defs/Speed"}],
                        "description": "The speed someone is walking at at the start of their walk.",
                    }
                },
                "required": ["early_speed"],
                "type": "object",
            },
            "name": "FormatResponse",
            "description": "Comment for how fast someone is walking",
        }
    ],
    "function_call": {"name": "FormatResponse"},
}

Error

in this specific case, the enum `Speed` does not seem to be respected as it returns `Slow` instead of `SLOW` and pydantic breaks.

Versions

this is comparing marvin 1.5.0 vs marvin 1.5.1 ( which now allows pydantic < 2.0 )

Additional context

No response

[aaazzam]

hey @bennnym we've been chatting on slack but wanted to bring more visibility here that we're actively working on this 🖖

[aaazzam]

hey @bennnym my bet is less on the ordering and it's more on $defs vs definitions in the ref_templates

have something cooking

[bennnym]

Oooh interesting. Look forward to testing it out. If you need me to look at some more complex tests just let me know the branch @aaazzam

[zzstoatzz]

hey @bennnym - do you still have the same problems on latest marvin / prefect? we've made some changes to the compat as of late

[bennnym]

I only use pydantic > 2.0

So I'm unable to comment on pydantic v1.

But all works as expected with v2.

[bennnym]

Hmm interesting that you made changes. As of a couple of days ago, I did have some regression where enums weren't returning their valid types and I encountered ValidationErrors consistently.

I had to handle this by adding a try except with a ai_classifier in the field_validator for pydantic.

[alexclaydon]

We're tracking something similar related to enums here on Instructor. The enum-description field isn't being populated. The timing is similar - just started happening a couple of days ago. It seems to occur even if you freeze the model at 0613. I note that the last Pydantic v2 release was over a month ago. Python 3.11.6.