Improper response with multiple output fields

[manankalra]

trafficstars

Issue: Reasoning is concatenated with any one of the output fields at random.

I have a signature with one input field and three output fields. Each of the fields have a small description. I'm using the COT module. The returned response is often imperfect and the reasoning string gets concatenated to either of the output fields. Is there a way to control this?

Example:

from dspy import InputField, OutputField, Signature
from dspy import ChainOfThought, Module, Prediction, Suggest


class SomeSignature(Signature):
    """Extract X, Y, and Z from something"""

    some_input = InputField(
        desc="some desc",
        prefix="some_input:",
    )
    X = OutputField(
        desc="some desc",
        prefix="X:",
    )
    Y = OutputField(
        desc="some desc",
        prefix="Y:",
    )
    Z = OutputField(
        desc="some desc",
        prefix="Z:",
    )



class SomeModule(Module):
    def __init__(self):
        super().__init__()
        self.extract = ChainOfThought(SomeSignature)

    def forward(self, some_input):
        result = self.extract(some_input=some_input)
        return Prediction(X=result.X, Y=result.Y, Z=result.Z)

Output:

• Sometimes the output is formatted perfectly:

Reasoning: ...
X: ...
Y: ...
Z: ...

• Most of the times it is either of the following and various other variations where the reasoning is suffixed with any of the output fields:

Reasoning: <blank>
X: <blank>
Y: <blank>
Z: .... <includes a long string that concatenates reasoning and output fields>


Reasoning: <blank>
Reasoning: ...
X: ...
Y: ...
Z: ... Reasoning: ...

[mikeedjones]

experimental=True might help with this, depending what is causing it :)

[okhat]

Yes, try dspy.configure(experimental=True) and ensure max_tokens is large enough for your use-case, e.g. 1024

[manankalra]

thanks, will give it a try and update here 👍🏼

[okhat]

Probably resolved now in DSPy 2.5 if you migrate https://github.com/stanfordnlp/dspy/blob/main/examples/migration.ipynb