How to handle fake messages that were not part of real coversations?

[simonw]

For Claude models a common pattern is to provide examples as a bunch of fake user/assistant messages:

curl https://api.anthropic.com/v1/messages \
     --header "x-api-key: $ANTHROPIC_API_KEY" \
     --header "anthropic-version: 2023-06-01" \
     --header "content-type: application/json" \
     --data \
'{
    "model": "claude-3-haiku-20240307",
    "max_tokens": 1024,
    "system": "You classify news stories as tech, opinion, world or usa",
    "messages": [
        {"role": "user", "content": "New Apple trackpad breeds nasty fungus"},
        {"role": "assistant", "content": "tech"},
        {"role": "user", "content": "I do not like how the world is going"},
        {"role": "assistant", "content": "opinion"},
        {"role": "user", "content": "Potential war brewing in Spain"},
        {"role": "assistant", "content": "world"},
        {"role": "user", "content": "Who will be the next senator from Vermont?"},
        {"role": "assistant", "content": "usa"},
        {"role": "user", "content": "Microsoft release Windows 334"}
    ]
}'

llm doesn't really support this yet - the conversation mechanism is just for when you reply to threads, but there isn't a clean way to feed it a whole bunch of "fake" conversation like this as part of a prompt.

[simonw]

One option could be a --raw mechanism something like this:

llm -m claude-3-haiku --raw '[
    {"role": "user", "content": "New Apple trackpad breeds nasty fungus"},
    {"role": "assistant", "content": "tech"},
    {"role": "user", "content": "I do not like how the world is going"},
    {"role": "assistant", "content": "opinion"},
    {"role": "user", "content": "Potential war brewing in Spain"},
    {"role": "assistant", "content": "world"},
    {"role": "user", "content": "Who will be the next senator from Vermont?"},
    {"role": "assistant", "content": "usa"},
    {"role": "user", "content": "Microsoft release Windows 334"}
]'

How would this be logged? Would these be turned into a conversation with a bunch of responses even though they never actually came back from the LLM itself?

[simonw]

Note that running the above and then following up with llm -c 'French economy set to boom' would need to trigger the full set of messages to be sent just as if they were stored as a real conversation.

[simonw]

Maybe an extra column on the responses table could record if a particular response was synthetic or actually came back from an LLM.