Boolean output failing when using TypedPredictor

[smathewvelocity]

Hi everyone

I tried to use the newly introduced TypedPredictor to return a boolean output, but it is always returning True, although, the inspect_history does reveal that the LLM gave the right result. Is this a bug?

from typing import List
import dspy

gpt4 = dspy.OpenAI(model="gpt-4-1106-preview", max_tokens=250)

dspy.settings.configure(lm=gpt4)


class JobQualifyModel(dspy.Signature):
    """
    Classify whether a candidate qualifies for a job
    Given, candidate's skills
    and, job name and description
    If there is insufficient information, the model should return False.
    """

    skills: List[str] = dspy.InputField(desc="List of candidate's skills")
    job_name: str = dspy.InputField(desc="Name of the job")
    job_description: str = dspy.InputField(desc="Description of the job")

    output: bool = dspy.OutputField(desc="True/False")


type_model = dspy.functional.TypedPredictor(JobQualifyModel)

job_model = dspy.ChainOfThought(JobQualifyModel)

job_name = "Event manager"
job_description = "Any adult can be accepted who does not write computer programs"
candidate_skills = ["coding", "marketing", "writing"]

res_job = job_model(
    skills=", ".join(candidate_skills),
    job_name=job_name,
    job_description=job_description,
)

res_job_type = type_model(
    skills=candidate_skills,
    job_name=job_name,
    job_description=job_description,
)

I have two issues:

1. How can I apply ChainOfThought reasoning using TypedPredictor?
2. Why is res_job_type.output always coming to be True?

The output of gpt4.inspect_history gives the right result.

> gpt4.inspect_history(n=1)

Classify whether a candidate qualifies for a job
    Given, candidate's skills
    and, job name and description
    If there is insufficient information, the model should return False.

---

Follow the following format.
Skills: List of candidate's skills
Job Name: Name of the job
Job Description: Description of the job
Output: True/False (Respond with a single bool value)
---

Skills:
[1] «coding»
[2] «marketing»
[3] «writing»

Job Name: Event manager
Job Description: Any adult can be accepted who does not write computer programs

Output: False

Thanks, Sam

[smathewvelocity]

I found dspy.TypedChainOfThought to solve the CoT issue. But still the output is always coming by default as True

Sam

[smathewvelocity]

Still facing issue. Updating the title so that it is clearer.

[qhoxie]

This is fixed in 2.4.9 (verified with a quick test of the code you posted). I believe this was the fix if you're curious: https://github.com/stanfordnlp/dspy/commit/982db444322cbf6a545a2504ce162b15dafee975

[smathewvelocity]

Thanks @qhoxie for letting me know. I indeed checked it and it works!