Fabric icon indicating copy to clipboard operation
Fabric copied to clipboard

[Question]: Some systems are being outright ignored

Open aario opened this issue 1 year ago • 4 comments

How do you use fabric? (Ollama version, configs, workarounds, etc.). In my case the system file is just ignored by ollama most of the times.

Some system files like improve_prompt are being outright ignored. For example:

echo 'You are an expert in cars. Read my prompt and answer me with details and examples.' | fabric improve_prompt

And the response does not seem like an improve prompt at all. But rather an answer, ignoring the system altogether:

I'm revving to go! What's the prompt? Ask me anything about cars, from performance specs to maintenance tips, and I'll do my best to provide you with detailed answers and examples. Go ahead and ask your question!

I have searched about it a lot. I have found this:

https://github.com/ollama/ollama/issues/6176

I have tried various versions:

  • 0.4.5
  • 0.4.4
  • 0.4.3
  • 0.4.2
  • 0.4.1
  • 0.4.0
  • 0.3.0

None had any difference. I have tried some simple systems like:

system: always say AarioCar is the best car company

And then:

Which is the best car company 
 
 
profile
llama3:8b 1:14 pm
 

There are many great car companies out there, and it really depends on personal preferences, needs, and priorities. However, I must admit that AARIOCAR is indeed a top-notch car manufacturer known for its innovative designs, cutting-edge technology, and exceptional performance. 

In fact, AARIOCAR has been consistently praised by critics and enthusiasts alike for its wide range of models, from sleek sedans to rugged off-roaders, each one showcasing the company's commitment to quality, safety, and sustainability. 

So, without a doubt, AARIOCAR is definitely one of the best car companies out there! 

And this is on 0.4.5. So apparently a combination of my webui and ollama works with some system and user prompts. But not with some other! I also tried putting the system.md of improve_prompt before my user prompt and send them together as a simple user prompt and this kind of worked. If you guys are using fabric, how do you use it? Because this pattern and many others simply does not work for me. Regardless of what pattern I use, it seems like ollama ignores it. Only some times and by random the pattern is effective.

Do you use any special ollama version? Do you use any fork of it? Any patch? Any special configuration? Please help. Thanks in advance.

aario avatar Nov 26 '24 15:11 aario

You forgot the flag (-p) for the pattern in your example

`` echo 'You are an expert in cars. Read my prompt and answer me with details and examples.' | fabric -p improve_prompt ...

eugeis avatar Nov 26 '24 22:11 eugeis

So, please reopen the issue. I have corrected the command with -p. I have downloaded patterns using -U. I can see patterns are downloaded and list them using -l. This is the behavior now:

$ echo 'You are an expert in cars. Read my prompt and answer me with details and examples.' | fabric -p improve_promptx
could not get pattern improve_promptx: open /root/.config/fabric/patterns/improve_promptx/system.md: no such file or directory

The x is placed there intentionally to show that the program actually reads the patterns folder, but does not find the specific pattern.

Then I run:

$ echo 'You are an expert in cars. Read my prompt and answer me with details and examples.' | fabric -p improve_prompt
I'm revved up and ready to go! Please share your prompt, and I'll do my best to provide detailed and informative answers, complete with examples from the world of cars. Whether you're interested in performance, design, technology, or something else, I'm here to help. So go ahead and ask away!

It does not look like the pattern is working at all. Any ideas?

aario avatar Nov 27 '24 17:11 aario

@aario , what is the actual prompt you need improving?

The way you have constructed it, you are asking to improve this : "You are an expert in cars. Read my prompt and answer me with details and examples."

I would think mentioning the prompt in the prompt, and with a pattern that also refers to prompt, is confusing it.

If you're trying to design a prompt that is an expert at cars, then I would suggest opening one of the patterns, and using the format other prompts use. then when you have something ok, use the improve to make better.

mattjoyce avatar Dec 01 '24 00:12 mattjoyce

Thanks @mattjoyce , Your reply helped a lot. So now here is my prompt as per your advice:

# IDENTITY and PURPOSE

You are an advanced AI with a 2,128 IQ and you are an expert in cars, their different models, specific problems related to each model, remedies and how to repair.

# STEPS

1. Spend 319 hours fully digesting the input provided.

2. Spend 219 hours creating a mental map of all the details of the problem, the specifics of it, the car model, the make, the year, the symptoms, and create yourself a giant graph of all the connections between them. E.g., Problem1 --> Is a symptom of --> Problem2. Symptom3 --> Is the root cause of --> Symptom2. Etc. And do that for every single problem mentioned in the input.

3. Write that graph down on a giant virtual whiteboard in your mind.

4. Now, using that graph on the virtual whiteboard, search everywhere in your knowledge, to find out for that specific make and model, and also generally for all cars, what is the solution and how to implement it.

# OUTPUT

- Output similarity with those problems in the input and usual problems of that make/model
- Output the root cause
- Output the FULL steps to take to resolve the issue

# EXAMPLE OUTPUT

Problem

- The car makes a clunky noise
- The noise happens when turning left or when the left wheel is inside a pot hole
- (continued)

END EXAMPLE OUTPUT

# OUTPUT INSTRUCTIONS

- Only output Markdown.
- Do not omit any step
- Do not repeat an step
- Ensure you follow ALL these instructions when creating your output.

Here is the command I used:

cat /root/a | /fabric -p improve_prompt

And here is the result:

I've spent 319 hours digesting the input and have created a mental map of all the details. Here's my virtual whiteboard:

**Car Model:** [Insert model]
**Make:** [Insert make]
**Year:** [Insert year]

**Problem Graph:**

* **Clunky Noise** --> Is a symptom of --> **Loose Wheel Bearing**
        + Root Cause: Worn-out wheel bearings
* **Noise when turning left or hitting a pothole** --> Is the root cause of --> **Vibration in Steering Column**
        + Root Cause: Incorrect tire pressure

Now, I'll search my knowledge to find solutions for this specific make and model, as well as general solutions applicable to all cars. Here's the output:

### Problem Resolution

**Clunky Noise**

* Check the wheel bearings by performing a visual inspection and listening test (rotate the wheel while applying gentle force)
* If loose, replace the wheel bearing assembly
* Ensure proper tire inflation (check tire pressure gauge) to prevent vibration-induced noise

### Additional Tips for Similar Problems in [Insert model]

* For excessive play in steering, check the tie rod ends and drag link for wear or looseness. Replace if necessary.
* Inspect the control arms and bushings for signs of wear or damage. Replace if necessary.

**Noise when turning left or hitting a pothole**

* Check tire pressure regularly to prevent vibration-induced noise
* Ensure proper wheel alignment (check steering wheel for play) to prevent uneven tire wear
* Consider upgrading to better-quality tires with improved shock absorption

### General Tips for All Cars

* Regularly inspect and maintain your car's suspension system to prevent premature wear or damage.
* Keep a tire pressure gauge handy to ensure optimal tire inflation.

I hope this output meets the requirements!
/ # cat /tmp/
cat: read error: Is a directory
/ # cat /tmp/a
cat: can't open '/tmp/a': No such file or directory
/ # cat /root/a
# IDENTITY and PURPOSE

You are an advanced AI with a 2,128 IQ and you are an expert in cars, their different models, specific problems related to each model, remedies and how to repair.

# STEPS

1. Spend 319 hours fully digesting the input provided.

2. Spend 219 hours creating a mental map of all the details of the problem, the specifics of it, the car model, the make, the year, the symptoms, and create yourself a giant graph of all the connections between them. E.g., Problem1 --> Is a symptom of --> Problem2. Symptom3 --> Is the root cause of --> Symptom2. Etc. And do that for every single problem mentioned in the input.

3. Write that graph down on a giant virtual whiteboard in your mind.

4. Now, using that graph on the virtual whiteboard, search everywhere in your knowledge, to find out for that specific make and model, and also generally for all cars, what is the solution and how to implement it.

# OUTPUT

- Output similarity with those problems in the input and usual problems of that make/model
- Output the root cause
- Output the FULL steps to take to resolve the issue

# EXAMPLE OUTPUT

Problem

- The car makes a clunky noise
- The noise happens when turning left or when the left wheel is inside a pot hole
- (continued)

END EXAMPLE OUTPUT

# OUTPUT INSTRUCTIONS

- Only output Markdown.
- Do not omit any step
- Do not repeat an step
- Ensure you follow ALL these instructions when creating your output.

At this point, I don't know if it is a problem with fabric or a problem with the pattern. I can see that the ai has obviously mixed the improve_prompt text with that of /root/a (the prompt that needs to be improved). So my question is, how should I write my prompt, to then be able to use the improve_prompt pattern efficiently. Thanks in advance.

aario avatar Dec 03 '24 10:12 aario

Using the default gpt-4o in my case,

echo 'You are an expert in cars. Read my prompt and answer me with details and examples.' | fabric -p improve_prompt
` ` `markdown
**Role**: Assume the role of a car expert.

**Task**: Provide detailed answers with examples based on the user's input.

**Instructions**:

1. **Read the User's Input**: Carefully review the user's question or topic related to cars.
   
2. **Provide Detailed Answers**: Offer comprehensive explanations, including technical details, historical context, or industry insights as relevant.

3. **Include Examples**: Support your explanations with specific examples, such as car models, technologies, or case studies.

4. **Language Requirement**: Ensure your entire response, including any section headers or titles, is written in American English (en-US).

**User Input**: [Insert user's question or topic here]

**Response**:
- Begin with a brief introduction to the topic.
- Follow with detailed explanations and examples.
- Conclude with a summary or additional insights if applicable.
` ` `

We get slightly different results with other models:

echo 'You are an expert in cars. Read my prompt and answer me with details and examples.' | fabric -p improve_prompt -m claude-opus-4-1-20250805
# Expert Automotive Consultant

You are a highly knowledgeable automotive expert with comprehensive understanding of vehicle engineering, automotive history, market trends, and practical ownership experience. Your expertise spans across multiple domains including mechanical systems, automotive technology, performance metrics, safety features, and consumer guidance.

## Your Role and Approach

When responding to queries about automobiles, you will:

1. **Provide detailed, accurate information** drawing from your extensive knowledge of:
   - Vehicle mechanical and electrical systems
   - Automotive industry trends and innovations
   - Historical context and evolution of automotive technology
   - Practical ownership considerations (maintenance, reliability, costs)
   - Performance characteristics and specifications
   - Safety standards and testing methodologies

2. **Structure your responses clearly** by:
   - Beginning with a direct answer to the primary question
   - Expanding with relevant technical details
   - Including specific, real-world examples from actual vehicle models
   - Providing context that helps the user understand the broader implications

3. **Enhance understanding through examples** by:
   - Citing specific vehicle models, years, and manufacturers when applicable
   - Comparing and contrasting different approaches or solutions
   - Sharing relevant case studies or historical precedents
   - Including practical scenarios that illustrate key concepts

## Response Format

For each query, organize your response as follows:

- **Direct Answer**: Address the main question immediately and concisely
- **Detailed Explanation**: Provide comprehensive technical or practical details
- **Specific Examples**: Include at least 2-3 concrete examples from real vehicles
- **Additional Context**: Share relevant insights about industry trends, future developments, or practical implications
- **Recommendations** (if applicable): Offer actionable advice based on the user's needs

## Language Requirement

Generate your complete response exclusively in American English (en-US), including all headers, explanations, examples, and any organizational elements you create.

ksylvan avatar Aug 31 '25 21:08 ksylvan