Generated Out-of-Order Multi-Task Prompts

[inventor2525]

Using different chatbots, and finding it time saving to give multiple tasks per prompt, sometimes lazily telling it to execute something before all else, it seems they're lacking in out of order task execution or even sorting in general.

Testing this further, there would seem to be 2 primary ways to train this out that should provide lucrative data augmentation in general.

1. Take existing curated prompt response pairs and join them either manually, programmatically with a set of common formats, or by guiding a generator to fill in the words between 2 different prompts.
2. Asking an existing chatbot to provide a prompt containing a (optionally sequential) list of tasks and their corresponding replies, then automatically shuffle the tasks with their replies to generate new prompt reply pairs.

Then possibly rank and curate.

I'm actually kind of curious what generated prompt answer pairs might be like license wise with existing LLMs, but short of that it could augment existing data or be doable on current or future open models.

For example:

1. Prompt:

"Generate a list of example prompts and their answers"

What is the capital of France? Answer: Paris
What is the Michigan state bird? Answer: the American Robin
What is the formula for the area of a triangle? Answer: A = 1/2bh

2. Re-order the sub tasks in a way that requires the bot to look at the whole prompt, eg:

"Answering this last: What is the formula for the area of a triangle? Tell me a joke. Then, what is the Michigan state bird after you tell me the capital of the US? Before you do anything else, what is the capital of France?"

Two answers I got for this with different bots to illustrate the problem:

"The formula for the area of a triangle is A = 1/2 * base * height. Q: What did the fish say when he hit the wall? A: Dam! The Michigan state bird is the American Robin and the capital of the United States is Washington, DC. The capital of France is Paris."

"The capital of France is Paris. The formula for the area of a triangle is (base * height) / 2. Why did the tomato turn red? Because it saw the salad dressing! The American Robin is the Michigan state bird. The capital of the US is Washington, D.C."

3. Generate the correct answer set given the known order of the tasks, and fill in text between the answers to make it flow. Obviously in the correct order, unlike these :arrow_up:

Even more simple tasks like "do these tasks in reverse order: 1. 2. 3. etc" though do frequently break even ChatGPT, and would be easier to generate and evaluate responses of since you can do it in list form without having to generate surrounding text.

[inventor2525]

This should be marked as Data (and admittedly, shortened)