Josh-XT
Enforce Formatting
Problem Description
Small models are having a hard time adhering to the formats we're trying to enforce in our prompts.
For example, when the LLM wants to execute commands, it is supposed to format like this
COMMANDS:
- command_name(arg1="some arg",arg2="some other arg")
However, some LLMs are listing them completely differently, or just listing what commands they want to run in a pseudo code summary that is wrong and/or not detected by the back end as commands to execute.
Proposed Solution
Add Guardrails if determined to be applicable and viable for the purposes of this project. https://github.com/ShreyaR/guardrails
Alternatives Considered
- Change format of prompts to force all inputs and outputs to be in JSON format, which may confuse the smaller models.
- Use a smaller model to validate formatting and reformat when the LLM didn't follow format?
Additional Context
Need to research further into Guardrails to see how it will impact the already small token limit.
Acknowledgements
- [X] My issue title is concise, descriptive, and in title casing.
- [X] I have searched the existing issues to make sure this feature has not been requested yet.
- [X] I have provided enough information for the maintainers to understand and evaluate this request.
@Josh-XT have you seen this one? https://github.com/1rgs/jsonformer
But i think people from guardrails repo will integrate with that. Saw some issues about that.
That definitely looks more my speed than Guardrails too. Everything JSON! I'll look more at that as the primary idea instead of guardrails.