fabric
fabric copied to clipboard
danielmiessler
[Feature request]: LocalLLM(ollama) support
What do you need?
this is awesome project, but it needs ollama support. the OpenAI api is the ease way out.
please add support for local LLMs too.
thank you
It has that support. Ollama and Claude. —model. For a list of models run —listmodels. If you have ollama running on your machine it will list the ollama models you have running.
i have ollama but not on this machine(its remote). i am looking to configure it as custom endpoint. is there a sample config?
I don’t have a custom endpoint coded in yet. I was looking through the ollama python library and I couldn’t find a way to do it. If anyone knows how to do this I’ll add it
check this out https://docs.crewai.com/how-to/LLM-Connections/#setting-up-ollama in crewai user can change the endpoint as global or per agent. please add support model change for fabric too
side quest. integration as tools for crewai to use fabric
I believe ollama support needs some refining:
- You cannot use
--listmodelsunless you provide an OpenAI key. There are various sources of models; I get that. If there is not a key for a source, skip it, provide a message - there is no key for the group - continue to the next source. - The model names returned from ollama are not correct; they are missing the last letter of their names. For some reason only llama2 returns the correct name.
Some enhancements:
- I would like to see the default model name reported from fabric if there is one. If not, return there is no default model.
- There is no command to get the version of fabric; Version is needed to report where the bugs are reported.
I used ollama v0.1.28, latest commit of fabric. Ollama has multiple models pulled:
codellama:latest 8fdf8f752f6e 3.8 GB 4 weeks ago
gemma:latest 430ed3535049 5.2 GB 2 weeks ago
llama2:latest 78e26419b446 3.8 GB 4 weeks ago
mistral:latest 61e88e884507 4.1 GB 6 weeks ago
neural-chat:latest 89fa737d3b85 4.1 GB 6 weeks ago
phi:latest e2fd6321a5fe 1.6 GB 4 weeks ago
qwen:latest d53d04290064 2.3 GB 4 weeks ago
Fabric reports these models:
Local Models:
codellam
gemm
llama2
mistr
neural-ch
phi
qwen
Looking forward for the next drop, George J.
Still does not work, additionally fabric does not distinguish between the variants of models (mistral for example). Please see below's output
NAME ID SIZE MODIFIED
codellama:latest 8fdf8f752f6e 3.8 GB 3 months ago
deepseek-coder:1.3b-base-q4_1 40fcd5fa2517 856 MB 3 months ago
llama2:latest fe938a131f40 3.8 GB 3 months ago
mistral:instruct 61e88e884507 4.1 GB 29 minutes ago
mistral:latest 1ab49bc0b6a8 4.1 GB 3 months ago
nomic-embed-text:latest 0a109f422b47 274 MB 2 weeks ago
(base) josh@M2-Pro:~$ fabric --listmodels
Failed to fetch models: HTTP 401
No OpenAI API key found. Please run fabric --setup and add the key if you wish to interact with openai
Failed to fetch models: HTTP 401
No OpenAI API key found. Please run fabric --setup and add the key if you wish to interact with openai
GPT Models:
Local Models:
codellama
deepseek-coder
llama2
mistral
mistral
nomic-embed-text
Claude Models:
claude-3-opus-20240229
To fix the output of the model regarding ollama line 298 needs to be modified on utils to say
fullOllamaList.append(model["name"]
instead of
fullOllamaList.append(model["name"].split(":")[0])
Ok I think I got it working with ollama models. Please see the below code from my working utils.py
import requests
import os
from openai import OpenAI
import asyncio
import pyperclip
import sys
import platform
from dotenv import load_dotenv
import zipfile
import tempfile
import re
import shutil
current_directory = os.path.dirname(os.path.realpath(__file__))
config_directory = os.path.expanduser("~/.config/fabric")
env_file = os.path.join(config_directory, ".env")
class Standalone:
def __init__(self, args, pattern="", env_file="~/.config/fabric/.env"):
"""Initialize the class with the provided arguments and environment file.
Args:
args: The arguments for initialization.
pattern: The pattern to be used (default is an empty string).
env_file: The path to the environment file (default is "~/.config/fabric/.env").
Returns:
None
Raises:
KeyError: If the "OPENAI_API_KEY" is not found in the environment variables.
FileNotFoundError: If no API key is found in the environment variables.
"""
# Expand the tilde to the full path
env_file = os.path.expanduser(env_file)
load_dotenv(env_file)
try:
apikey = os.environ["OPENAI_API_KEY"]
self.client = OpenAI()
self.client.api_key = apikey
except:
print("No API key found. Use the --apikey option to set the key")
self.local = False
self.config_pattern_directory = config_directory
self.pattern = pattern
self.args = args
self.model = args.model
self.claude = False
sorted_gpt_models, ollamaList, claudeList = self.fetch_available_models()
self.local = self.model.strip() in ollamaList
self.claude = self.model.strip() in claudeList
async def localChat(self, messages):
from ollama import AsyncClient
response = await AsyncClient().chat(model=self.model, messages=messages)
print(response["message"]["content"])
async def localStream(self, messages):
from ollama import AsyncClient
async for part in await AsyncClient().chat(
model=self.model, messages=messages, stream=True
):
print(part["message"]["content"], end="", flush=True)
async def claudeStream(self, system, user):
from anthropic import AsyncAnthropic
self.claudeApiKey = os.environ["CLAUDE_API_KEY"]
Streamingclient = AsyncAnthropic(api_key=self.claudeApiKey)
async with Streamingclient.messages.stream(
max_tokens=4096,
system=system,
messages=[user],
model=self.model,
temperature=0.0,
top_p=1.0,
) as stream:
async for text in stream.text_stream:
print(text, end="", flush=True)
print()
message = await stream.get_final_message()
async def claudeChat(self, system, user):
from anthropic import Anthropic
self.claudeApiKey = os.environ["CLAUDE_API_KEY"]
client = Anthropic(api_key=self.claudeApiKey)
message = client.messages.create(
max_tokens=4096,
system=system,
messages=[user],
model=self.model,
temperature=0.0,
top_p=1.0,
)
print(message.content[0].text)
def streamMessage(self, input_data: str, context=""):
"""Stream a message and handle exceptions.
Args:
input_data (str): The input data for the message.
Returns:
None: If the pattern is not found.
Raises:
FileNotFoundError: If the pattern file is not found.
"""
wisdomFilePath = os.path.join(
config_directory, f"patterns/{self.pattern}/system.md"
)
user_message = {"role": "user", "content": f"{input_data}"}
wisdom_File = os.path.join(current_directory, wisdomFilePath)
system = ""
buffer = ""
if self.pattern:
try:
with open(wisdom_File, "r") as f:
if context:
system = context + "\n\n" + f.read()
else:
system = f.read()
system_message = {"role": "system", "content": system}
messages = [system_message, user_message]
except FileNotFoundError:
print("pattern not found")
return
else:
if context:
messages = [{"role": "system", "content": context}, user_message]
else:
messages = [user_message]
try:
if self.local:
asyncio.run(self.localStream(messages))
elif self.claude:
from anthropic import AsyncAnthropic
asyncio.run(self.claudeStream(system, user_message))
else:
stream = self.client.chat.completions.create(
model=self.model,
messages=messages,
temperature=0.0,
top_p=1,
frequency_penalty=0.1,
presence_penalty=0.1,
stream=True,
)
for chunk in stream:
if chunk.choices[0].delta.content is not None:
char = chunk.choices[0].delta.content
buffer += char
if char not in ["\n", " "]:
print(char, end="")
elif char == " ":
print(" ", end="") # Explicitly handle spaces
elif char == "\n":
print() # Handle newlines
sys.stdout.flush()
except Exception as e:
if "All connection attempts failed" in str(e):
print(
"Error: cannot connect to llama2. If you have not already, please visit https://ollama.com for installation instructions"
)
if "CLAUDE_API_KEY" in str(e):
print(
"Error: CLAUDE_API_KEY not found in environment variables. Please run --setup and add the key"
)
if "overloaded_error" in str(e):
print(
"Error: Fabric is working fine, but claude is overloaded. Please try again later."
)
else:
print(f"Error: {e}")
print(e)
if self.args.copy:
pyperclip.copy(buffer)
if self.args.output:
with open(self.args.output, "w") as f:
f.write(buffer)
def sendMessage(self, input_data: str, context=""):
"""Send a message using the input data and generate a response.
Args:
input_data (str): The input data to be sent as a message.
Returns:
None
Raises:
FileNotFoundError: If the specified pattern file is not found.
"""
wisdomFilePath = os.path.join(
config_directory, f"patterns/{self.pattern}/system.md"
)
user_message = {"role": "user", "content": f"{input_data}"}
wisdom_File = os.path.join(current_directory, wisdomFilePath)
system = ""
if self.pattern:
try:
with open(wisdom_File, "r") as f:
if context:
system = context + "\n\n" + f.read()
else:
system = f.read()
system_message = {"role": "system", "content": system}
messages = [system_message, user_message]
except FileNotFoundError:
print("pattern not found")
return
else:
if context:
messages = [{"role": "system", "content": context}, user_message]
else:
messages = [user_message]
try:
if self.local:
asyncio.run(self.localChat(messages))
elif self.claude:
asyncio.run(self.claudeChat(system, user_message))
else:
response = self.client.chat.completions.create(
model=self.model,
messages=messages,
temperature=0.0,
top_p=1,
frequency_penalty=0.1,
presence_penalty=0.1,
)
print(response.choices[0].message.content)
except Exception as e:
if "All connection attempts failed" in str(e):
print(
"Error: cannot connect to llama2. If you have not already, please visit https://ollama.com for installation instructions"
)
if "CLAUDE_API_KEY" in str(e):
print(
"Error: CLAUDE_API_KEY not found in environment variables. Please run --setup and add the key"
)
if "overloaded_error" in str(e):
print(
"Error: Fabric is working fine, but claude is overloaded. Please try again later."
)
if "Attempted to call a sync iterator on an async stream" in str(e):
print(
"Error: There is a problem connecting fabric with your local ollama installation. Please visit https://ollama.com for installation instructions. It is possible that you have chosen the wrong model. Please run fabric --listmodels to see the available models and choose the right one with fabric --model <model> or fabric --changeDefaultModel. If this does not work. Restart your computer (always a good idea) and try again. If you are still having problems, please visit https://ollama.com for installation instructions."
)
else:
print(f"Error: {e}")
print(e)
if self.args.copy:
pyperclip.copy(response.choices[0].message.content)
if self.args.output:
with open(self.args.output, "w") as f:
f.write(response.choices[0].message.content)
def fetch_available_models(self):
gptlist = []
fullOllamaList = []
claudeList = ["claude-3-opus-20240229"]
try:
headers = {"Authorization": f"Bearer {self.client.api_key}"}
response = requests.get("https://api.openai.com/v1/models", headers=headers)
if response.status_code == 200:
models = response.json().get("data", [])
# Filter only gpt models
gpt_models = [
model for model in models if model.get("id", "").startswith(("gpt"))
]
# Sort the models alphabetically by their ID
sorted_gpt_models = sorted(gpt_models, key=lambda x: x.get("id"))
for model in sorted_gpt_models:
gptlist.append(model.get("id"))
else:
print(f"Failed to fetch models: HTTP {response.status_code}")
pass
except:
print(
"No OpenAI API key found. Please run fabric --setup and add the key if you wish to interact with openai"
)
import ollama
try:
default_modelollamaList = ollama.list()["models"]
for model in default_modelollamaList:
fullOllamaList.append(model["name"])
except:
fullOllamaList = []
return gptlist, fullOllamaList, claudeList
def get_cli_input(self):
"""aided by ChatGPT; uses platform library
accepts either piped input or console input
from either Windows or Linux
Args:
none
Returns:
string from either user or pipe
"""
system = platform.system()
if system == "Windows":
if not sys.stdin.isatty(): # Check if input is being piped
return sys.stdin.read().strip() # Read piped input
else:
# Prompt user for input from console
return input("Enter Question: ")
else:
return sys.stdin.read()
class Update:
def __init__(self):
"""Initialize the object with default values."""
self.repo_zip_url = (
"https://github.com/danielmiessler/fabric/archive/refs/heads/main.zip"
)
self.config_directory = os.path.expanduser("~/.config/fabric")
self.pattern_directory = os.path.join(self.config_directory, "patterns")
os.makedirs(self.pattern_directory, exist_ok=True)
print("Updating patterns...")
self.update_patterns() # Start the update process immediately
def update_patterns(self):
"""Update the patterns by downloading the zip from GitHub and extracting it."""
with tempfile.TemporaryDirectory() as temp_dir:
zip_path = os.path.join(temp_dir, "repo.zip")
self.download_zip(self.repo_zip_url, zip_path)
extracted_folder_path = self.extract_zip(zip_path, temp_dir)
# The patterns folder will be inside "fabric-main" after extraction
patterns_source_path = os.path.join(
extracted_folder_path, "fabric-main", "patterns"
)
if os.path.exists(patterns_source_path):
# If the patterns directory already exists, remove it before copying over the new one
if os.path.exists(self.pattern_directory):
shutil.rmtree(self.pattern_directory)
shutil.copytree(patterns_source_path, self.pattern_directory)
print("Patterns updated successfully.")
else:
print("Patterns folder not found in the downloaded zip.")
def download_zip(self, url, save_path):
"""Download the zip file from the specified URL."""
response = requests.get(url)
response.raise_for_status() # Check if the download was successful
with open(save_path, "wb") as f:
f.write(response.content)
print("Downloaded zip file successfully.")
def extract_zip(self, zip_path, extract_to):
"""Extract the zip file to the specified directory."""
with zipfile.ZipFile(zip_path, "r") as zip_ref:
zip_ref.extractall(extract_to)
print("Extracted zip file successfully.")
return extract_to # Return the path to the extracted contents
class Alias:
def __init__(self):
self.config_files = []
home_directory = os.path.expanduser("~")
self.patterns = os.path.join(home_directory, ".config/fabric/patterns")
if os.path.exists(
os.path.join(home_directory, ".config/fabric/fabric-bootstrap.inc")
):
self.config_files.append(
os.path.join(home_directory, ".config/fabric/fabric-bootstrap.inc")
)
self.remove_all_patterns()
self.add_patterns()
print("Aliases added successfully. Please restart your terminal to use them.")
def add(self, name, alias):
for file in self.config_files:
with open(file, "a") as f:
f.write(f"alias {name}='{alias}'\n")
def remove(self, pattern):
for file in self.config_files:
# Read the whole file first
with open(file, "r") as f:
wholeFile = f.read()
# Determine if the line to be removed is in the file
target_line = f"alias {pattern}='fabric --pattern {pattern}'\n"
if target_line in wholeFile:
# If the line exists, replace it with nothing (remove it)
wholeFile = wholeFile.replace(target_line, "")
# Write the modified content back to the file
with open(file, "w") as f:
f.write(wholeFile)
def remove_all_patterns(self):
allPatterns = os.listdir(self.patterns)
for pattern in allPatterns:
self.remove(pattern)
def find_line(self, name):
for file in self.config_files:
with open(file, "r") as f:
lines = f.readlines()
for line in lines:
if line.strip("\n") == f"alias ${name}='{alias}'":
return line
def add_patterns(self):
allPatterns = os.listdir(self.patterns)
for pattern in allPatterns:
self.add(pattern, f"fabric --pattern {pattern}")
class Setup:
def __init__(self):
"""Initialize the object.
Raises:
OSError: If there is an error in creating the pattern directory.
"""
self.config_directory = os.path.expanduser("~/.config/fabric")
self.pattern_directory = os.path.join(self.config_directory, "patterns")
os.makedirs(self.pattern_directory, exist_ok=True)
self.env_file = os.path.join(self.config_directory, ".env")
self.gptlist = []
self.fullOllamaList = []
self.claudeList = ["claude-3-opus-20240229"]
load_dotenv(self.env_file)
try:
openaiapikey = os.environ["OPENAI_API_KEY"]
self.openaiapi_key = openaiapikey
except:
pass
try:
self.fetch_available_models()
except:
pass
def fetch_available_models(self):
headers = {"Authorization": f"Bearer {self.openaiapi_key}"}
response = requests.get("https://api.openai.com/v1/models", headers=headers)
if response.status_code == 200:
models = response.json().get("data", [])
# Filter only gpt models
gpt_models = [
model for model in models if model.get("id", "").startswith(("gpt"))
]
# Sort the models alphabetically by their ID
sorted_gpt_models = sorted(gpt_models, key=lambda x: x.get("id"))
for model in sorted_gpt_models:
self.gptlist.append(model.get("id"))
else:
print(f"Failed to fetch models: HTTP {response.status_code}")
pass
import ollama
try:
default_modelollamaList = ollama.list()["models"]
for model in default_modelollamaList:
self.fullOllamaList.append(model["name"])
except:
self.fullOllamaList = []
allmodels = self.gptlist + self.fullOllamaList + self.claudeList
return allmodels
def api_key(self, api_key):
"""Set the OpenAI API key in the environment file.
Args:
api_key (str): The API key to be set.
Returns:
None
Raises:
OSError: If the environment file does not exist or cannot be accessed.
"""
api_key = api_key.strip()
if not os.path.exists(self.env_file) and api_key:
with open(self.env_file, "w") as f:
f.write(f"OPENAI_API_KEY={api_key}")
print(f"OpenAI API key set to {api_key}")
elif api_key:
# erase the line OPENAI_API_KEY=key and write the new key
with open(self.env_file, "r") as f:
lines = f.readlines()
with open(self.env_file, "w") as f:
for line in lines:
if "OPENAI_API_KEY" not in line:
f.write(line)
f.write(f"OPENAI_API_KEY={api_key}")
def claude_key(self, claude_key):
"""Set the Claude API key in the environment file.
Args:
claude_key (str): The API key to be set.
Returns:
None
Raises:
OSError: If the environment file does not exist or cannot be accessed.
"""
claude_key = claude_key.strip()
if os.path.exists(self.env_file) and claude_key:
with open(self.env_file, "r") as f:
lines = f.readlines()
with open(self.env_file, "w") as f:
for line in lines:
if "CLAUDE_API_KEY" not in line:
f.write(line)
f.write(f"CLAUDE_API_KEY={claude_key}")
elif claude_key:
with open(self.env_file, "w") as f:
f.write(f"CLAUDE_API_KEY={claude_key}")
def update_fabric_command(self, line, model):
fabric_command_regex = re.compile(
r"(alias.*fabric --pattern\s+\S+.*?)( --model.*)?'"
)
match = fabric_command_regex.search(line)
if match:
base_command = match.group(1)
# Provide a default value for current_flag
current_flag = match.group(2) if match.group(2) else ""
new_flag = ""
new_flag = f" --model {model}"
# Update the command if the new flag is different or to remove an existing flag.
# Ensure to add the closing quote that was part of the original regex
return f"{base_command}{new_flag}'\n"
else:
return line # Return the line unmodified if no match is found.
def update_fabric_alias(self, line, model):
fabric_alias_regex = re.compile(r"(alias fabric='[^']+?)( --model.*)?'")
match = fabric_alias_regex.search(line)
if match:
base_command, current_flag = match.groups()
new_flag = f" --model {model}"
# Update the alias if the new flag is different or to remove an existing flag.
return f"{base_command}{new_flag}'\n"
else:
return line # Return the line unmodified if no match is found.
def clear_alias(self, line):
fabric_command_regex = re.compile(r"(alias fabric='[^']+?)( --model.*)?'")
match = fabric_command_regex.search(line)
if match:
base_command = match.group(1)
return f"{base_command}'\n"
else:
return line # Return the line unmodified if no match is found.
def clear_env_line(self, line):
fabric_command_regex = re.compile(
r"(alias.*fabric --pattern\s+\S+.*?)( --model.*)?'"
)
match = fabric_command_regex.search(line)
if match:
base_command = match.group(1)
return f"{base_command}'\n"
else:
return line # Return the line unmodified if no match is found.
def pattern(self, line):
fabric_command_regex = re.compile(r"(alias fabric='[^']+?)( --model.*)?'")
match = fabric_command_regex.search(line)
if match:
base_command = match.group(1)
return f"{base_command}'\n"
else:
return line # Return the line unmodified if no match is found.
def clean_env(self):
"""Clear the DEFAULT_MODEL from the environment file.
Returns:
None
"""
user_home = os.path.expanduser("~")
sh_config = None
# Check for shell configuration files
if os.path.exists(
os.path.join(user_home, ".config/fabric/fabric-bootstrap.inc")
):
sh_config = os.path.join(user_home, ".config/fabric/fabric-bootstrap.inc")
else:
print("No environment file found.")
if sh_config:
with open(sh_config, "r") as f:
lines = f.readlines()
with open(sh_config, "w") as f:
for line in lines:
modified_line = line
# Update existing fabric commands
if "fabric --pattern" in line:
modified_line = self.clear_env_line(modified_line)
elif "fabric=" in line:
modified_line = self.clear_alias(modified_line)
f.write(modified_line)
self.remove_duplicates(env_file)
else:
print("No shell configuration file found.")
def default_model(self, model):
"""Set the default model in the environment file.
Args:
model (str): The model to be set.
"""
model = model.strip()
if model:
# Write or update the DEFAULT_MODEL in env_file
allModels = self.claudeList + self.fullOllamaList + self.gptlist
if model not in allModels:
print(
f"Error: {model} is not a valid model. Please run fabric --listmodels to see the available models."
)
sys.exit()
# Compile regular expressions outside of the loop for efficiency
user_home = os.path.expanduser("~")
sh_config = None
# Check for shell configuration files
if os.path.exists(
os.path.join(user_home, ".config/fabric/fabric-bootstrap.inc")
):
sh_config = os.path.join(user_home, ".config/fabric/fabric-bootstrap.inc")
if sh_config:
with open(sh_config, "r") as f:
lines = f.readlines()
with open(sh_config, "w") as f:
for line in lines:
modified_line = line
# Update existing fabric commands
if "fabric --pattern" in line:
modified_line = self.update_fabric_command(modified_line, model)
elif "fabric=" in line:
modified_line = self.update_fabric_alias(modified_line, model)
f.write(modified_line)
print(
f"""Default model changed to {
model}. Please restart your terminal to use it."""
)
else:
print("No shell configuration file found.")
def remove_duplicates(self, filename):
unique_lines = set()
with open(filename, "r") as file:
lines = file.readlines()
with open(filename, "w") as file:
for line in lines:
if line not in unique_lines:
file.write(line)
unique_lines.add(line)
def patterns(self):
"""Method to update patterns and exit the system.
Returns:
None
"""
Update()
def run(self):
"""Execute the Fabric program.
This method prompts the user for their OpenAI API key, sets the API key in the Fabric object, and then calls the patterns method.
Returns:
None
"""
print("Welcome to Fabric. Let's get started.")
apikey = input(
"Please enter your OpenAI API key. If you do not have one or if you have already entered it, press enter.\n"
)
self.api_key(apikey.strip())
print(
"Please enter your claude API key. If you do not have one, or if you have already entered it, press enter.\n"
)
claudekey = input()
self.claude_key(claudekey.strip())
self.patterns()
class Transcribe:
def youtube(video_id):
"""
This method gets the transciption
of a YouTube video designated with the video_id
Input:
the video id specifing a YouTube video
an example url for a video: https://www.youtube.com/watch?v=vF-MQmVxnCs&t=306s
the video id is vF-MQmVxnCs&t=306s
Output:
a transcript for the video
Raises:
an exception and prints error
"""
try:
transcript_list = YouTubeTranscriptApi.get_transcript(video_id)
transcript = ""
for segment in transcript_list:
transcript += segment["text"] + " "
return transcript.strip()
except Exception as e:
print("Error:", e)
return None
class AgentSetup:
def apiKeys(self):
"""Method to set the API keys in the environment file.
Returns:
None
"""
print("Welcome to Fabric. Let's get started.")
browserless = input("Please enter your Browserless API key\n")
serper = input("Please enter your Serper API key\n")
# Entries to be added
browserless_entry = f"BROWSERLESS_API_KEY={browserless}"
serper_entry = f"SERPER_API_KEY={serper}"
# Check and write to the file
with open(env_file, "r+") as f:
content = f.read()
# Determine if the file ends with a newline
if content.endswith("\n"):
# If it ends with a newline, we directly write the new entries
f.write(f"{browserless_entry}\n{serper_entry}\n")
else:
# If it does not end with a newline, add one before the new entries
f.write(f"\n{browserless_entry}\n{serper_entry}\n")
Thanks! good looking out. I fixed it. Make sure it works for you and let me know
it can use different Ollama location (not just the default localhost and port) if we change AsyncClient() to something like this: AsyncClient(host=os.environ["OLLAMA_HOST"]) In the setup we could ask for OLLAMA_HOST and put in .env (the default could be the default ollama url -> http://localhost:11434) What do you think?
... i try to integrate Fabric with Slack as a Bot, using local Ollama with local LLM in different docker containers
Well currently I am using the ollama python library, and I can't find a way to change the default ollama url in that library in the documentation. If you can figure that out I'll incorporate it
https://github.com/ollama/ollama-python/blob/main/ollama/_client.py
class AsyncClient(BaseClient):
def __init__(self, host: Optional[str] = None, **kwargs) -> None:
super().__init__(httpx.AsyncClient, host, **kwargs)
"host" is an optional paramteter of AsyncClient
OK I added it as --remoteOllamaServer. Please test it and make sure it works and I didn't break anything :)
Seems it works. My steps are as below:
##I am not sure if the latest "--model qwen" is needed, seems not.
$ fabric --setup --remoteOllamaServer http://127.0.0.1:11434 --model qwen
Welcome to Fabric. Let's get started.
Please enter your OpenAI API key. If you do not have one or if you have already entered it, press enter.
there is no key for the group
OpenAI API key set to there is no key for the group
Please enter your claude API key. If you do not have one, or if you have already entered it, press enter.
there is no key for the group
Please enter your YouTube API key. If you do not have one, or if you have already entered it, press enter.
there is no key for the group
Updating patterns...
Downloaded zip file successfully.
Extracted zip file successfully.
Patterns updated successfully.
Aliases added successfully. Please restart your terminal to use them.
# fabric --listmodels
Failed to fetch models: HTTP 401
No OpenAI API key found. Please run fabric --setup and add the key if you wish to interact with openai
Failed to fetch models: HTTP 401
No OpenAI API key found. Please run fabric --setup and add the key if you wish to interact with openai
GPT Models:
Local Models:
codellama:latest
gemma:latest
mistral:latest
qwen:latest
Claude Models:
claude-3-opus-20240229
claude-3-sonnet-20240229
claude-2.1
cat t.txt | fabric --model qwen:latest --pattern summarize
Failed to fetch models: HTTP 401
No OpenAI API key found. Please run fabric --setup and add the key if you wish to interact with openai
<--it output the summary info correctly here-->
the update was very fast. thank you
a few improvement notes there should be a ~/fabric.conf file with contents of end point addresses for ollama endpoints
#--remoteOllamaServer
default="http://localhost:11434" #default path
#extra path's
s1="http://127.0.0.2:11434"
s2="http://127.0.0.3:11434"
s3="http://127.0.0.4:11434"
and so on....
and a System wide env variable for ~/fabric.conf file location, perhaps user needs a different path for it, can setup like this
FABRIC_CONF="/root/hidden-location/fabric.conf"
server names (s1) should be defined by user
fabric automatically look for this file and read it as ollama endpoints
in terminal calling fabric like this list models from s1 ollama
fabric s1 --listmodels or fabric --s1 --listmodels
if s1 or --s1 is not called it should default to local ollama or GPT4 etc...
and one last thing
fabric --listmodels
should list models from other ollama paths too. :)
i think this is the best way to approach this, what do you guys think
Something not work, what am I missed? the machine can reach the ollama: root@5be46746d531:/# curl http://172.17.0.2:11434/api/generate -d '{"model": "mistral", "prompt": "How much is the fish?"}' ...
But i cannot see my local models: root@5be46746d531:/# fabric --listmodels --remoteOllamaServer http://172.17.0.2:11434 Failed to fetch models: HTTP 401 No OpenAI API key found. Please run fabric --setup and add the key if you wish to interact with openai Failed to fetch models: HTTP 401 No OpenAI API key found. Please run fabric --setup and add the key if you wish to interact with openai GPT Models:
Local Models:
Claude Models: claude-3-opus-20240229 claude-3-sonnet-20240229 claude-2.1
I guess instead of this: ... import ollama try: default_modelollamaList = ollama.list()['models'] ... This would be better if host is exist (if --remoteOllamaServer is set): from ollama import Client client = Client(host=host) client.list....
to be openai compatible endpoint on ollama it still needs key try this in your curl request "OPENAI_API_KEY": "NA" it should get around error,
question for the dev. does fabric uses langchain tools openchat for openai api calling or its from openai package?
I run Fabric on windows WSL, and run Ollama on native windows. "> fabric --listmodel" don't seem to list any models from ollama.
How do I setup Fabric to access LLM loaded via Ollama in this case? 127.0.0.1 doesn't seem to work, as wsl and navtive windows seem to act as two different system/network.
thanks, Sean
I run Fabric on windows WSL, and run Ollama on native windows. "> fabric --listmodel" don't seem to list any models from ollama.
How do I setup Fabric to access LLM loaded via Ollama in this case? 127.0.0.1 doesn't seem to work, as wsl and navtive windows seem to act as two different system/network.
thanks, Sean
run ollama on WSL much more stable, windows version is buggy,
I run Fabric on windows WSL, and run Ollama on native windows. "> fabric --listmodel" don't seem to list any models from ollama. How do I setup Fabric to access LLM loaded via Ollama in this case? 127.0.0.1 doesn't seem to work, as wsl and navtive windows seem to act as two different system/network. thanks, Sean
run ollama on WSL much more stable, windows version is buggy,
I am sure running both ollama and fabric on WSL would work out. I am also running Open/Ollama WebUI on native windows via docker desktop. I am not sure the dockered-webui would work with WSL-ollama. I will just stay with OAI for fabric now, until the fabric supports native windows.
Since Ollama is OAI compatible and there are tons of OAI-compatible API-services out there, like groq, together, etc. I suggest, using the defacto industry standard (ATM) OAI library only. but like in crewai and other, make a portfolio of models and endpoints available. a lot of AI-apps integrate litellm.ai as library, or one can use it as proxy/router/cache, etc.
as @rony432 has suggested:
a few improvement notes there should be a ~/fabric.conf file with contents of end point addresses for ollama endpoints
#--remoteOllamaServer default="http://localhost:11434" #default path #extra path's s1="http://127.0.0.2:11434" s2="http://127.0.0.3:11434" s3="http://127.0.0.4:11434"
one could predefine models by usecases, i.e. translate: HOST_1:port_1 model A math: HOST_1:port_2 model B coder: HOST_2:port_1 model C visual: HOST_3:port_1 model D …
Hello,
Firstly, congratulation for this project that seems to be very interessting.
Unfortunatly I can't make it work with Ollama. I have a local Ollama service with Gemma that I can curl with no issue :
curl http://localhost:11434/api/generate -d '{
"model": "gemma",
"prompt": "Why is the sky blue?"
}'
{"model":"gemma","created_at":"2024-03-20T09:29:56.609134562Z","response":"The","done":false}
....
But when I try to configure fabric :
fabric --setup --remoteOllamaServer http://localhost:11434
It ask me for OpenAI, Claude and YT API but did nothing more :
Welcome to Fabric. Let's get started.
Please enter your OpenAI API key. If you do not have one or if you have already entered it, press enter.
Please enter your claude API key. If you do not have one, or if you have already entered it, press enter.
Please enter your YouTube API key. If you do not have one, or if you have already entered it, press enter.
Updating patterns...
Downloaded zip file successfully.
Extracted zip file successfully.
Patterns updated successfully.
If I then ask for model list :
fabric --listmodels
Please run --setup to set up your API key and download patterns.
Even with this command :
fabric --listmodels --remoteOllamaServer http://localhost:11434
Please run --setup to set up your API key and download patterns.
In fact every request send me back to the setup :
fabric --list
Please run --setup to set up your API key and download patterns.
There may certainly be something I didn't do right, but I have no clue for the moment. Thank you
Just run --setup first without the remote server flag. Fabric --setup". If you don't want to use OpenAI or Claude just click enter. Once you do that that message will go away
I should have mention that I did that first. But as it didn't work and I saw in this discussion that it's the command to configure the Ollama server I thought I did worng in the first place :
fabric$ fabric --setup
Welcome to Fabric. Let's get started.
Please enter your OpenAI API key. If you do not have one or if you have already entered it, press enter.
Please enter your claude API key. If you do not have one, or if you have already entered it, press enter.
Please enter your YouTube API key. If you do not have one, or if you have already entered it, press enter.
Updating patterns...
Downloaded zip file successfully.
Extracted zip file successfully.
Patterns updated successfully.
fabric$ fabric --listmodels
Please run --setup to set up your API key and download patterns.
fabric$ curl http://localhost:11434/api/generate -d '{
"model": "gemma",
"prompt": "Why is the sky blue?"
}'
{"model":"gemma","created_at":"2024-03-20T10:14:58.21685401Z","response":"The","done":false}
{"model":"gemma","created_at":"2024-03-20T10:14:58.459595578Z","response":" sky","done":false}
I have exactly the same problem as @Papoulos.
After
fabric --setup
Skipping keys for OpenAI, Claude and YB, I tried the following:
# fabric --remoteOllamaServer http://127.0.0.1:11434 --listmodels
GPT Models:
gpt-3.5-turbo
gpt-3.5-turbo-0125
gpt-3.5-turbo-0301
gpt-3.5-turbo-0613
gpt-3.5-turbo-1106
gpt-3.5-turbo-16k
gpt-3.5-turbo-16k-0613
gpt-3.5-turbo-instruct
gpt-3.5-turbo-instruct-0914
Local Models:
Claude Models:
claude-3-opus-20240229
claude-3-sonnet-20240229
claude-3-haiku-20240307
claude-2.1
Also:
# cat Boeing-clean.txt | fabric --remoteOllamaServer http://127.0.0.1:11434 --model llama2 -sp write_essay
Error: Error code: 404 - {'error': {'message': 'The model `llama2` does not exist', 'type': 'invalid_request_error', 'param': None, 'code': 'model_not_found'}}
Error code: 404 - {'error': {'message': 'The model `llama2` does not exist', 'type': 'invalid_request_error', 'param': None, 'code': 'model_not_found'}}
A curl request from another console works fine:
$ curl http://127.0.0.1:11434/api/generate -d '{ "model": "llama2", "prompt": "Why is the sky blue?"}'
{"model":"llama2","created_at":"2024-03-20T16:29:54.952782624Z","response":"\n","done":false}
{"model":"llama2","created_at":"2024-03-20T16:29:55.306774474Z","response":"The","done":false}
Incidentally, it would be nice to have an option to skip updating patterns when we use fabric --setup. While we try to figure out how to make this work it is just annoying having to wait for these updates that we know won't yield anything we care about.
I try the same command as you did but still got the message for configuration :
fabric$ cat ../test.txt | fabric --remoteOllamaServer http://127.0.0.1:11434 --model gemma -sp write_essay
Please run --setup to set up your API key and download patterns.
I'm also running both fabric (latest from git commit c3df1e7ecac5bd5b854b91f7b244e441945aeb81 ) - and ollama 0.1.29 on Windows 11:
PS C:\Users\kayvan\src\fabric> ollama -v
ollama version is 0.1.29
PS C:\Users\kayvan\src\fabric> ollama list
NAME ID SIZE MODIFIED
llama2:latest 78e26419b446 3.8 GB 3 weeks ago
llava:latest 8dd30f6b0cb1 4.7 GB 2 days ago
mistral:latest 61e88e884507 4.1 GB 3 days ago
mixtral:instruct 7708c059a8bb 26 GB 3 days ago
mixtral:latest 7708c059a8bb 26 GB 3 weeks ago
And it's working correctly:
PS C:\Users\kayvan\src\fabric> Invoke-WebRequest -Uri http://localhost:11434
StatusCode : 200
StatusDescription : OK
Content : Ollama is running
RawContent : HTTP/1.1 200 OK
Date: Thu, 21 Mar 2024 22:10:17 GMT
Content-Type: text/plain; charset=utf-8
Content-Length: 17
Ollama is running
Headers : {[Date, System.String[]], [Content-Type, System.String[]], [Content-Length, System.String[]]}
Images : {}
InputFields : {}
Links : {}
RawContentLength : 17
RelationLink : {}
And here is the local model respnding:
PS C:\Users\kayvan\src\fabric> (Invoke-WebRequest -method POST -Body '{"model":"mistral", "prompt":"Why is the sky blue?", "stream": false}' -uri http://localhost:11434/api/generate ).Content | ConvertFrom-json
model : mistral
created_at : 3/21/2024 10:10:53 PM
response : The color of the sky appears blue due to a process called scattering. When the Sun emits light, it sends out electromagnetic waves in all parts of the spectrum. However, Earth's atmosphere scatters
short-wavelength light (blue and violet) more effectively than longer wavelengths (yellow, orange, and red). As a result, when we look up at the sky, we predominantly see the blue light that has been
scattered in all directions. Additionally, some of the violet light gets absorbed by the ozone layer in the stratosphere, leaving us with more blue than violet.
However, it's important to note that the color of the sky isn't always blue - during sunrise or sunset, for instance, the sky can take on hues of pink, orange, and red due to the presence of more scattered
longer wavelengths.
done : True
context : {733, 16289, 28793, 28705…}
total_duration : 4894802400
load_duration : 3076050200
prompt_eval_count : 15
prompt_eval_duration : 16438000
eval_count : 187
eval_duration : 1802114000
However, fabric still does not see the local models:
PS C:\Users\kayvan\src\fabric> fabric --listmodels
GPT Models:
gpt-3.5-turbo
gpt-3.5-turbo-0125
gpt-3.5-turbo-0301
gpt-3.5-turbo-0613
gpt-3.5-turbo-1106
gpt-3.5-turbo-16k
gpt-3.5-turbo-16k-0613
gpt-3.5-turbo-instruct
gpt-3.5-turbo-instruct-0914
gpt-4
gpt-4-0125-preview
gpt-4-0613
gpt-4-1106-preview
gpt-4-turbo-preview
gpt-4-vision-preview
Local Models:
Claude Models:
claude-3-opus-20240229
claude-3-sonnet-20240229
claude-3-haiku-20240307
claude-2.1
I'm really wondering if you guys just aren't missing the .config/fabric/.env file & fabric isn't setup to access Window's user file path as its Mac/Linux based. From what I understant, the .config/fabric dir gets created when you run that setup stuff but as Windows doesn't recognize ~ has a user's home dir it just may not be able to read work correctly.
This is all just a theory though! I did see the author of fabric say Mac/Linux/WSL are the only officially supported distributions atm. I hope I'm wrong though. ;)
