PoC freeform output w/ XML inserts (instead of strict JSON)

[Pwuts]

Duplicates

• [X] I have searched the existing issues

Summary 💡

LLMs are bad at generating JSON. Not surprising, because compared to natural language it's like Brainfuck. It's not human-readable without whitespace. Enter XML: it shares syntax with all of the web, and the opening and closing tags make it more readable (and thus more writable too).

Credits to @ntindle for the idea!

Examples 🌈

• User: I'm curious about countries’ capitals. Like, what's the capital of the U.S.?

• Assistant: I don't know what the capital of the U.S. is. I'll look for this through a web search.

  <use_tool>
    <name>web_search</name>
    <args>
      <question>What is the capital of the U.S.?</question>
    </args>
  </use_tool>
  

• Function: The capital of the U.S. is Washington, D.C.

• Assistant: Let's also feed your curiosity by searching for capitals of 10 more countries!

  <use_tool>
    <name>web_search</name>
    <args>
      <question>Top 10 country capitals</question>
    </args>
  </use_tool>
  

• Function: [insert 10 capitals here]

• Assistant: <answer>The capital of the U.S. is Washington, D.C. Some other national capitals are: [insert 10 capitals here]</answer>

Motivation 🔦

Reasoning: The closer our output format is to the "natural" format that the LLM is trained to output, the higher the reliability and performance can be.

[zedatrix]

What about considering markdown? It's even closer to natural language

[ntindle]

I considered it but structuring nested markdown seemed non-trivial to identify key value responses when nested and in a list.

Would love an example of the outputs above how you’d imagine them in markdown

[zedatrix]

I think there are some quasi markdown formats out there that might be able to help with that e.g.slack has a modified version for their UI builder. I can research and respond with what you asked.

[github-actions[bot]]

This issue has automatically been marked as stale because it has not had any activity in the last 50 days. You can unstale it by commenting or removing the label. Otherwise, this issue will be closed in 10 days.

[Pwuts]

Unstale! 🪄

[MKdir98]

Did someone start this issue? Do we need to change promts? I'm working with mistral. I was wondering can I start this one and if tests get passed it shows everything is ok? Or this is more than this?

[Pwuts]

@MKdir98 a PoC for this issue needs changes to both the prompt and the parsing stage. For example you could try converting the OneShotPromptStrategy in autogpt/agents/prompt_strategies/one_shot.py to use XML instead of JSON. Then if that works in the most basic form, you could try specifying different types of cells (e.g. use_tool, python, answer) and see if the LLM uses them correctly.

[github-actions[bot]]

This issue has automatically been marked as stale because it has not had any activity in the last 50 days. You can unstale it by commenting or removing the label. Otherwise, this issue will be closed in 10 days.