Prompt improvement: Scrubbed self-critique to save tokens.

[lc0rp]

trafficstars

Background

The LLM context window is today's dial-up modem. LLM short-term memory is a precious resource that the community is working hard to optimize.

While prevailing research suggests that reflection can improve LLM accuracy, similar research suggests that asking the LLM to self-critique within the same request is ineffective. Instead, multiple requests should be made to self-criticize correctly, which involves feeding the output (and, optionally, the critique) back into the model. We are not doing this today.

Better zero-shot prompts are:

• Let's think step by step
• Let's work this output in a step-by-step way to be sure we have the right answer

REF:

1. https://arxiv.org/abs/2305.02897.pdf (Automatically discovered COT - (Konstantin et al., 2003)
2. https://arxiv.org/abs/2205.11916 (LLMs are zero-shot reasoners - "Let's think step by step." (Kojima et al., 2022))
3. https://arxiv.org/abs/2211.01910 (LLMs are human-level prompt engineers - "Let's work through this..." (Zhou et al., 2023))

Changes

This PR scrubs self-critique from the current prompts as a precursor to adding some of the proposed zero-shot improvements.

Documentation

No changes.

Test Plan

The entire existing test suite has been run to confirm no regressions have been introduced. No tests have been added because current tests cover the changed parts.

PR Quality Checklist

• [X] My pull request is atomic and focuses on a single change.
• [X] I have thoroughly tested my changes with multiple prompts.
• [X] I have considered potential risks and mitigations for my changes.
• [X] I have documented my changes clearly and comprehensively.
• [X] I have not snuck in any "extra" minor tweaks or changes

[vercel[bot]]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Comments	Updated (UTC)
docs	⬜️ Ignored (Inspect)			May 9, 2023 6:56pm

[github-actions[bot]]

This pull request has conflicts with the base branch, please resolve those so we can evaluate the pull request.

[jimmylegendary]

I want to test this PR. Please assign me

[jimmylegendary]

I want to test this PR. Please assign me

I can't test this pr due to merge conflicts, so it needs to be fixed first