danielmiessler
[Bug]: Input text is included in both System and User context
What happened?
With the default settings I'm getting the original content that is piped to fabric included in both the system prompt and the user prompt, this can be seen when using the --dry-run option:
echo "This is a test" | fabric -p my_test --dry-run
Dry run: Would send the following request:
System:
You are highly skilled document summarization AI.
# OUTPUT:
Summarize the content in a single paragraph with no more than 100 words.
# INPUT:
this is a test
User:
this is a test
Options:
Model: llama3.2:latest
Temperature: 0.700000
TopP: 0.900000
PresencePenalty: 0.000000
FrequencyPenalty: 0.000000
empty response
Expected output should only include the input this is a test in the User content, and not repeated in the System prompt.
I've found that the duplication of the user content in the system prompt leads to very poor prompt adherence, and also has the side effect of doubling required model context window size.
Cause of the issue seems to be here in the pattern applyVariables which is forcing the {{input}} to be included. Commenting out this block fixes the problem, but breaks the --raw option which does need the pattern and input combined.
Version check
- [X] Yes I was.
Relevant log output
No response
Relevant screenshots (optional)
No response
oh, I can replicate.
@scross01 {{input}} is appended if it's not in the pattern, and then replaced with the actual user input. This was implemented to allow user input to be moved to other locations in a pattern, such as at the front, or encapsulated in tags .
The previous behaviour was to simply append the pattern with the user input. So i think the problem is elsewhere. I will have a look.
For reference this is the prompt file system.md. There is no {{input}} variable in the prompt content
You are highly skilled document summarization AI.
# OUTPUT:
Summarize the content in a single paragraph with no more than 100 words.
# INPUT:
@mattjoyce I'm still getting the input content in both the system and user prompt.
Installed the latest from source
$ go install github.com/danielmiessler/fabric@latest
$ fabric --version
v1.4.123
$ echo "This is a another test" | fabric -p my_summarize --dry-run
Dry run: Would send the following request:
System:
You are highly skilled document summarization AI.
# OUTPUT:
Summarize the content in a single paragraph with no more than 100 words.
# INPUT:
This is a another test
User:
This is a another test
Options:
Model: llama3.2:3b
Temperature: 0.700000
TopP: 0.900000
PresencePenalty: 0.000000
FrequencyPenalty: 0.000000
empty response
@scross01 I cannot replicate this.
~ ./fabric --version
v1.4.124
~ echo "This is a another test" | fabric -p ai --dry-run
Dry run: Would send the following request:
System:
# IDENTITY and PURPOSE
You are an expert at interpreting the heart and spirit of a question and answering in an insightful manner.
# STEPS
- Deeply understand what's being asked.
- Create a full mental model of the input and the question on a virtual whiteboard in your mind.
- Answer the question in 3-5 Markdown bullets of 10 words each.
# OUTPUT INSTRUCTIONS
- Only output Markdown bullets.
- Do not output warnings or notes—just the requested sections.
# INPUT:
INPUT:
User:
This is a another test
Options:
Model: gpt-4o
Temperature: 0.700000
TopP: 0.900000
PresencePenalty: 0.000000
FrequencyPenalty: 0.000000
empty response
Nope, sorry to say it's still not working for me. I've tried using two methods of installing just to double check, and get the same result with both
$ go install github.com/danielmiessler/fabric@latest
go: downloading github.com/danielmiessler/fabric v1.4.124
$ fabric --version
v1.4.124
$ git clone https://github.com/danielmiessler/fabric.git
$ cd fabric
$ go build .
$ ./fabric --version
At the time of testing
git log
commit 6d00405eb66e7c232e8c558407587d190fa4ed13 (HEAD -> main, tag: v1.4.124, origin/main, origin/HEAD)
Author: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Date: Sat Dec 21 14:01:47 2024 +0000
Update version to v1.4.124 and commit
Result from both installs:
$ echo "this is a test message with v1.4.124" | ./fabric --model llama3.2:3b -p ai --dry-run
Dry run: Would send the following request:
System:
# IDENTITY and PURPOSE
You are an expert at interpreting the heart and spirit of a question and answering in an insightful manner.
# STEPS
- Deeply understand what's being asked.
- Create a full mental model of the input and the question on a virtual whiteboard in your mind.
- Answer the question in 3-5 Markdown bullets of 10 words each.
# OUTPUT INSTRUCTIONS
- Only output Markdown bullets.
- Do not output warnings or notes—just the requested sections.
# INPUT:
INPUT:
this is a test message with v1.4.124
User:
this is a test message with v1.4.124
Options:
Model: llama3.2:3b
Temperature: 0.700000
TopP: 0.900000
PresencePenalty: 0.000000
FrequencyPenalty: 0.000000
empty respons
@scross01 sorry, you were sort of right to begin with, it is cause by the implementation of the template system. Previously the the system message was system(context, patten(vars)) user(input) assistant(response)
What we have now is
system(context, pattern (vars, input (vars)))) user(input(vars)) assistant(response)
So my proposal is to structure the initial messages like this.
@eugeis this is a departure from the original fabric message structure, but I think it will not have any detrimental effect. Are you supportive?
My testing works fine.
Hi @mattjoyce, thank you for the great visualization!
I believe the system role plays a much more significant part in certain models. While it might work fine for a single message and response, in the context of session-based conversations, we risk losing the clarity and impact of the initial prompt.
What are your thoughts?
I think we should not change it. Keep what has been working, and put the user input as user role message.
It mean we cannot move user input around, but folk can use vars and plugins to achieve same results with a little effort.
I will need to adjust chatter.go and pattern.go to build the session like this.
Apologies, I had to take a break through January, I will pick this back up soon.
Actually I thought this was resolved. I could not replicate, but I will pick it back up and verify.
@ksylvan @eugeis I think the problem is, I provided a way to have patterns where the input is not always the last think appended to the pattern. This was in response to prompting guidelines from Anthropic (at the time) recommending xml encapsulation.
you can do this <EXAMPLE> {{input}} </EXAMPLE>
and the user input (piped or cli) will be substituted. if not {{input}} is found in the pattern it will just add one, causing user input to be appended.
this causes a double up. so I think I should remove that capability, and if folk want to do that they can use variables.
Sorry for the noise, @mattjoyce but I can not reproduce this either, so I think unless someone comes back with a clear reproducible case, I think we can close this and the other issue that duplicates it.
I'm still able to reproduce
fabric --version
v1.4.187
echo "This is a test" | fabric -p summarize --dry-run
Dry run: Would send the following request:
System:
# IDENTITY and PURPOSE
You are an expert content summarizer. You take content in and output a Markdown formatted summary using the format below.
Take a deep breath and think step by step about how to best accomplish this goal using the following steps.
# OUTPUT SECTIONS
- Combine all of your understanding of the content into a single, 20-word sentence in a section called ONE SENTENCE SUMMARY:.
- Output the 10 most important points of the content as a list with no more than 15 words per point into a section called MAIN POINTS:.
- Output a list of the 5 best takeaways from the content in a section called TAKEAWAYS:.
# OUTPUT INSTRUCTIONS
- Create the output using the formatting above.
- You only output human readable Markdown.
- Output numbered lists, not bullets.
- Do not output warnings or notes—just the requested sections.
- Do not repeat items in the output sections.
- Do not start items with the same opening words.
# INPUT:
INPUT:
This is a test
User:
This is a test
The 'This is a test' message appears twice in the output, in both the System and User messages.
EDIT: changed above example use one of the standard patterns.
No change noted:
$ echo "Ancient Egypt and Ancient Mesopotamia" | compare_and_contrast --dry-run
Dry run: Would send the following request:
System:
# IDENTITY and PURPOSE
Please be brief. Compare and contrast the list of items.
# STEPS
Compare and contrast the list of items
# OUTPUT INSTRUCTIONS
Please put it into a markdown table.
Items along the left and topics along the top.
# INPUT:
INPUT:
Ancient Egypt and Ancient Mesopotamia
User:
Ancient Egypt and Ancient Mesopotamia
Options:
Model: gpt-4o
Temperature: 0.700000
TopP: 0.900000
PresencePenalty: 0.000000
FrequencyPenalty: 0.000000
Also:
$ fabric --version
v1.4.187
Thank you @cosmicaug and @scross01 - I'll look again a bit later today.
@ksylvan @eugeis I think the problem is, I provided a way to have patterns where the input is not always the last think appended to the pattern. This was in response to prompting guidelines from Anthropic (at the time) recommending xml encapsulation.
my words you can do this {{input}}
and the user input (piped or cli) will be substituted. if not {{input}} is found in the pattern it will just add one, causing user input to be appended.
this causes a double up. so I think I should remove that capability, and if folk want to do that they can use variables.
Would this proposed change cause any unintended consequences or surprises for users?
I have not looked at this change (I wouldn't be the best person to), but can you check to see what it does when used in conjunction with the --raw option?
It occurred to me that it might be desirable to see how it is working currently when that is in effect. I am seeing that the bug manifests when the --raw option is in use as well.
My guess is that if you fix the one you will fix the other but it needs to be confirmed.
$ echo "Ancient Egypt and Ancient Mesopotamia" | compare_and_contrast --raw --dry-run
Dry run: Would send the following request:
User:
# IDENTITY and PURPOSE
Please be brief. Compare and contrast the list of items.
# STEPS
Compare and contrast the list of items
# OUTPUT INSTRUCTIONS
Please put it into a markdown table.
Items along the left and topics along the top.
# INPUT:
INPUT:
Ancient Egypt and Ancient Mesopotamia
Ancient Egypt and Ancient Mesopotamia
Options:
Model: gpt-4o
Temperature: 0.700000
TopP: 0.900000
PresencePenalty: 0.000000
FrequencyPenalty: 0.000000
From the description given when using fabric --help, I would have expected the pattern to be shifted to the user prompt with the piped in input appended to it. Instead we see that that the former happens correctly; but the latter happens with the same duplication that we saw before.
@cosmicaug I'm looking at the raw mode interactions now... Draft PR in process: https://github.com/danielmiessler/fabric/pull/1474
I needed to fix some weirdness with the Anthropic client code. Now testing it with all the providers I have access to.
Okay, @cosmicaug - Can you test this? https://github.com/danielmiessler/fabric/pull/1474
When we get confirmation from you guys, we can close this issue too. @eugeis merged the PR already. Thank you!
The simple test cases that I used seem to be correct.
According to what is shown when using the --dry-run option, the behavior with the --raw option works as I understand that it should.
$ echo "Ancient Egypt and Ancient Mesopotamia" | compare_and_contrast --raw --dry-run
Dry run: Would send the following request:
User:
# IDENTITY and PURPOSE
Please be brief. Compare and contrast the list of items.
# STEPS
Compare and contrast the list of items
# OUTPUT INSTRUCTIONS
Please put it into a markdown table.
Items along the left and topics along the top.
# INPUT:
INPUT:
Ancient Egypt and Ancient Mesopotamia\n\nAncient Egypt and Ancient Mesopotamia
Options:
Model: gpt-4o
Temperature: 0.700000
TopP: 0.900000
PresencePenalty: 0.000000
FrequencyPenalty: 0.000000
empty response
The behavior without the --raw option also seems to work fine:
$ echo "Ancient Egypt and Ancient Mesopotamia" | compare_and_contrast --dry-run
Dry run: Would send the following request:
System:
# IDENTITY and PURPOSE
Please be brief. Compare and contrast the list of items.
# STEPS
Compare and contrast the list of items
# OUTPUT INSTRUCTIONS
Please put it into a markdown table.
Items along the left and topics along the top.
# INPUT:
INPUT:
Ancient Egypt and Ancient Mesopotamia
Options:
Model: gpt-4o
Temperature: 0.700000
TopP: 0.900000
PresencePenalty: 0.000000
FrequencyPenalty: 0.000000
empty response
The former has no system prompt side and the latter has no user prompt side which, in my understanding, is as it should be.
For a sampling of actually trying to do the thing, actually sending the normal and raw cases to gpt-4o, gemini-2.5-flash-preview-04-17 & claude-3-opus-latest instead of dry running them, produced results as I expected.
I just pulled the latest updates on main. Now I'm getting duplicate input when the --raw option is used.
INPUT:
This is a test\n\nThis is a test
With out the raw option the output is correct.
INPUT:
This is a test
Full output:
echo "This is a test" | fabric -p summarize --dry-run --raw
Dry run: Would send the following request:
User:
# IDENTITY and PURPOSE
You are an expert content summarizer. You take content in and output a Markdown formatted summary using the format below.
Take a deep breath and think step by step about how to best accomplish this goal using the following steps.
# OUTPUT SECTIONS
- Combine all of your understanding of the content into a single, 20-word sentence in a section called ONE SENTENCE SUMMARY:.
- Output the 10 most important points of the content as a list with no more than 15 words per point into a section called MAIN POINTS:.
- Output a list of the 5 best takeaways from the content in a section called TAKEAWAYS:.
# OUTPUT INSTRUCTIONS
- Create the output using the formatting above.
- You only output human readable Markdown.
- Output numbered lists, not bullets.
- Do not output warnings or notes—just the requested sections.
- Do not repeat items in the output sections.
- Do not start items with the same opening words.
# INPUT:
INPUT:
This is a test\n\nThis is a test
fabric --version
v1.4.189
@cosmicaug same is in your sample output:
$ echo "Ancient Egypt and Ancient Mesopotamia" | compare_and_contrast --raw --dry-run
...
INPUT:
Ancient Egypt and Ancient Mesopotamia\n\nAncient Egypt and Ancient Mesopotamia
...