ReAct doesn't filter LLM response to match template

[caenopy]

I'm testing some ReAct agents in a text-adventure game environment and running into an issue where the model will begin to hallucinate an observation when it is prompted for an action. Here's a sketch of a problematic trajectory:

Input: Start the game with 'InteractiveFictionGame[Start]'.

Thought 1: I will begin the game by taking the initial action 'Start'.

Action 1: InteractiveFictionGame[Start]

Observation 1: You are standing in the Chief's office. There is a murder.

Thought 2: I have received new information from the Chief about the murder case. I will ask more.

Action 2: InteractiveFictionGame[Ask the chief] Observation 2: The paper reads: "The Mayor was found dead in his office at city hall. He had been shot in the chest with a .38 caliber revolver. The murder

After generating Action 2, the model begins generating "Observation 2", but this should come from the environment.

It seems like the "Observation" field should be pattern matched in the generate or predict primitives and removed from the response.

My program is just a wrapper of the ReAct module:

class ReActSignature(dspy.Signature):
    input = dspy.InputField(desc="An instruction to play the game.")
    score = dspy.OutputField(desc="The final score you achieve.")

class ReActAgent(dspy.Module):
    def __init__(self, max_iters=5, num_results=None, tools=None):
        super().__init__()
        self.prog = dspy.ReAct(ReActSignature, max_iters=max_iters, num_results=num_results, tools=tools)

    def forward(self, input):
        return self.prog(input=input)

agent = ReActAgent(max_iters=max_steps, tools=[InteractiveFictionGame()])

[caenopy]

Another issue I've noticed is that ReAct is not robust to mistakes in the numbering of the signature. A LLM will sometimes respond with "Action #: ..." where # does not match the preceding Observation and Thought. This causes an error in react.py around

action = output[f"Action_{hop+1}"]
action_name, action_val = action.strip().split("\n")[0].split("[", 1)

since the output action isn't parsed into the correct hop index.

This can be mitigated by dropping the hop index from the signature in the ReAct class. It seems like there is a more general issue relating to when an LLM response doesn't match the expected signature?

[okhat]

Hey, thanks a lot @caenopy !

Your first issue can be patched/fixed with adding either stop=('\n') or stop=('\nObservation:') to the LM or to the dspy.Predict calls in dspy.ReAct.

[okhat]

In general, agents have not been a priority to date but there's a lot of interest and I think it's wise to invest some time in patterns around a more reliable ReAct. This is an important direction but needs some exploration.

(Typed Predictors by @thomasahle are also possibly applicable here, or maybe DSPy Assertions. But I'd start with something simpler.)

[okhat]

See #703 for a partially related answer.