TypedChainOfThought Errors (too many retries, json output should start and end with { and }'))

[Dyke-F]

I am trying to structure very simple medical information (python lists of strings, on average not longer than 5 items) using TypedChainOfThought.

Frequently run into: ValueError: ('Too many retries trying to get the correct output format. Try simplifying the requirements.', {'result': "ValueError('json output should start and end with { and }')"})

Although llm.inspect_history shows that the model does it right:

Reasoning: Let's think step by step in order to Reasoning: Let's think step by step in order to produce the result. We need to assign unique integers to each criterion. Since each line in the provided criteria list represents a separate criterion and there are no nested criteria, each line should be assigned a unique integer.

1. The first criterion is "Prior treatment with an agent targeting KRAS G12C (e.g., AMG 510, Sotorasib)." This will be assigned the integer 0.
2. The second criterion is "Active brain metastases." This will be assigned the integer 1.

Result: Valid json object with a python list of integers having the same length as criteria, where each integer represents a unique criterion. Must be of length 2. Respond with a single JSON object.

{"value": [0, 1]}

(which is valid json)

=== The Signature is: class CriteriaAssignmentSignature(dspy.Signature): ....... (ommited for brevity here) .... Think step by step, before assigning numbers to each criterion, try to identify each subtopic even if they are nested. Example output: json {"value": [0, 1, 2, 3]} If the criteria list only contains a single element, return: json {"value": [0]} """ criteria: List[str] = dspy.InputField(desc="List of line by line split trial inclusion or exclusion criteria.") result: conlist(int, min_length=len(_criteria), max_length=len(_criteria)) = dspy.OutputField(desc=f"Valid json object with a python list of integers having the same length as criteria, where each integer represents a unique criterion. Must be of length {len(_criteria)}") # type: ignore

What am i missing here? Even Chatgpt-4o solves these issues easily. Trying with GPT-4o via API here.