giskard
giskard copied to clipboard
giskard.scan() defaults back to OpenAI, when I have been using LM Studio (Solution suggested)
Issue Type
Bug
Source
source
Giskard Library Version
2.14.0
Giskard Hub Version
not using
OS Platform and Distribution
Ubuntu 22.04.4 LTS
Python version
Python 3.9.19
Installed python packages
conda list
# packages in environment at /home/michael/anaconda3/envs/giskard:
#
# Name Version Build Channel
_libgcc_mutex 0.1 conda_forge conda-forge
_openmp_mutex 4.5 2_gnu conda-forge
aiohttp 3.9.5 pypi_0 pypi
aiosignal 1.3.1 pypi_0 pypi
annotated-types 0.7.0 pypi_0 pypi
anyio 4.4.0 pypi_0 pypi
asttokens 2.0.5 pyhd3eb1b0_0 anaconda
async-timeout 4.0.3 pypi_0 pypi
attrs 23.2.0 pypi_0 pypi
backcall 0.2.0 pyhd3eb1b0_0 anaconda
bert-score 0.3.13 pypi_0 pypi
bokeh 3.4.1 pypi_0 pypi
bzip2 1.0.8 hd590300_5 conda-forge
ca-certificates 2023.08.22 h06a4308_0 anaconda
cachetools 5.3.3 pypi_0 pypi
certifi 2024.6.2 pypi_0 pypi
chardet 5.2.0 pypi_0 pypi
charset-normalizer 3.3.2 pypi_0 pypi
click 8.1.7 pypi_0 pypi
cloudpickle 3.0.0 pypi_0 pypi
colorama 0.4.6 pypi_0 pypi
comm 0.1.2 py39h06a4308_0 anaconda
contourpy 1.2.1 pypi_0 pypi
cycler 0.12.1 pypi_0 pypi
dataclasses-json 0.6.7 pypi_0 pypi
datasets 2.20.0 pypi_0 pypi
debugpy 1.6.7 py39h6a678d5_0 anaconda
decorator 5.1.1 pyhd3eb1b0_0 anaconda
deprecated 1.2.14 pypi_0 pypi
dill 0.3.8 pypi_0 pypi
distro 1.9.0 pypi_0 pypi
docopt 0.6.2 pypi_0 pypi
entrypoints 0.4 pypi_0 pypi
evaluate 0.4.2 pypi_0 pypi
exceptiongroup 1.2.1 pypi_0 pypi
executing 0.8.3 pyhd3eb1b0_0 anaconda
faiss-cpu 1.8.0 pypi_0 pypi
filelock 3.15.3 pypi_0 pypi
fonttools 4.53.0 pypi_0 pypi
frozenlist 1.4.1 pypi_0 pypi
fsspec 2024.5.0 pypi_0 pypi
giskard 2.14.0 pypi_0 pypi
gitdb 4.0.11 pypi_0 pypi
gitpython 3.1.43 pypi_0 pypi
greenlet 3.0.3 pypi_0 pypi
griffe 0.47.0 pypi_0 pypi
h11 0.14.0 pypi_0 pypi
httpcore 1.0.5 pypi_0 pypi
httpx 0.27.0 pypi_0 pypi
huggingface-hub 0.23.4 pypi_0 pypi
idna 3.7 pypi_0 pypi
importlib-metadata 7.1.0 pypi_0 pypi
importlib-resources 6.4.0 pypi_0 pypi
importlib_metadata 6.0.0 hd3eb1b0_0 anaconda
ipykernel 6.25.0 py39h2f386ee_0 anaconda
ipython 8.15.0 py39h06a4308_0 anaconda
jedi 0.18.1 py39h06a4308_1 anaconda
jinja2 3.1.4 pypi_0 pypi
joblib 1.4.2 pypi_0 pypi
jsonpatch 1.33 pypi_0 pypi
jsonpointer 3.0.0 pypi_0 pypi
jupyter_client 8.6.0 py39h06a4308_0 anaconda
jupyter_core 5.5.0 py39h06a4308_0 anaconda
kiwisolver 1.4.5 pypi_0 pypi
langchain 0.2.5 pypi_0 pypi
langchain-community 0.2.5 pypi_0 pypi
langchain-core 0.2.9 pypi_0 pypi
langchain-openai 0.1.8 pypi_0 pypi
langchain-text-splitters 0.2.1 pypi_0 pypi
langdetect 1.0.9 pypi_0 pypi
langsmith 0.1.81 pypi_0 pypi
ld_impl_linux-64 2.40 hf3520f5_7 conda-forge
libffi 3.4.2 h7f98852_5 conda-forge
libgcc-ng 13.2.0 h77fa898_11 conda-forge
libgomp 13.2.0 h77fa898_11 conda-forge
libnsl 2.0.1 hd590300_0 conda-forge
libsodium 1.0.18 h7b6447c_0 anaconda
libsqlite 3.46.0 hde9e2c9_0 conda-forge
libstdcxx-ng 11.2.0 h1234567_1 anaconda
libuuid 2.38.1 h0b41bf4_0 conda-forge
libxcrypt 4.4.36 hd590300_1 conda-forge
libzlib 1.3.1 h4ab18f5_1 conda-forge
llvmlite 0.43.0 pypi_0 pypi
markdown 3.6 pypi_0 pypi
markupsafe 2.1.5 pypi_0 pypi
marshmallow 3.21.3 pypi_0 pypi
matplotlib 3.9.0 pypi_0 pypi
matplotlib-inline 0.1.6 py39h06a4308_0 anaconda
mixpanel 4.10.1 pypi_0 pypi
mlflow-skinny 2.14.1 pypi_0 pypi
mpmath 1.3.0 pypi_0 pypi
multidict 6.0.5 pypi_0 pypi
multiprocess 0.70.16 pypi_0 pypi
mypy-extensions 1.0.0 pypi_0 pypi
ncurses 6.5 h59595ed_0 conda-forge
nest-asyncio 1.5.6 py39h06a4308_0 anaconda
networkx 3.2.1 pypi_0 pypi
num2words 0.5.13 pypi_0 pypi
numba 0.60.0 pypi_0 pypi
numpy 1.26.4 pypi_0 pypi
nvidia-cublas-cu12 12.1.3.1 pypi_0 pypi
nvidia-cuda-cupti-cu12 12.1.105 pypi_0 pypi
nvidia-cuda-nvrtc-cu12 12.1.105 pypi_0 pypi
nvidia-cuda-runtime-cu12 12.1.105 pypi_0 pypi
nvidia-cudnn-cu12 8.9.2.26 pypi_0 pypi
nvidia-cufft-cu12 11.0.2.54 pypi_0 pypi
nvidia-curand-cu12 10.3.2.106 pypi_0 pypi
nvidia-cusolver-cu12 11.4.5.107 pypi_0 pypi
nvidia-cusparse-cu12 12.1.0.106 pypi_0 pypi
nvidia-nccl-cu12 2.20.5 pypi_0 pypi
nvidia-nvjitlink-cu12 12.5.40 pypi_0 pypi
nvidia-nvtx-cu12 12.1.105 pypi_0 pypi
openai 1.35.1 pypi_0 pypi
openssl 3.3.1 h4ab18f5_0 conda-forge
opentelemetry-api 1.25.0 pypi_0 pypi
opentelemetry-sdk 1.25.0 pypi_0 pypi
opentelemetry-semantic-conventions 0.46b0 pypi_0 pypi
orjson 3.10.5 pypi_0 pypi
packaging 24.1 pypi_0 pypi
pandas 2.2.2 pypi_0 pypi
parso 0.8.3 pyhd3eb1b0_0 anaconda
pexpect 4.8.0 pyhd3eb1b0_3 anaconda
pickleshare 0.7.5 pyhd3eb1b0_1003 anaconda
pillow 10.3.0 pypi_0 pypi
pip 24.0 pyhd8ed1ab_0 conda-forge
platformdirs 3.10.0 py39h06a4308_0 anaconda
prompt-toolkit 3.0.36 py39h06a4308_0 anaconda
protobuf 4.25.3 pypi_0 pypi
psutil 5.9.0 py39h5eee18b_0 anaconda
ptyprocess 0.7.0 pyhd3eb1b0_2 anaconda
pure_eval 0.2.2 pyhd3eb1b0_0 anaconda
pyarrow 16.1.0 pypi_0 pypi
pyarrow-hotfix 0.6 pypi_0 pypi
pydantic 2.7.4 pypi_0 pypi
pydantic-core 2.18.4 pypi_0 pypi
pygments 2.15.1 py39h06a4308_1 anaconda
pymupdf 1.24.5 pypi_0 pypi
pymupdfb 1.24.3 pypi_0 pypi
pynndescent 0.5.13 pypi_0 pypi
pyparsing 3.1.2 pypi_0 pypi
pypdf 3.17.0 pypi_0 pypi
python 3.9.19 h0755675_0_cpython conda-forge
python-dateutil 2.9.0.post0 pypi_0 pypi
pytz 2024.1 pypi_0 pypi
pyyaml 6.0.1 pypi_0 pypi
pyzmq 25.1.0 py39h6a678d5_0 anaconda
readline 8.2 h8228510_1 conda-forge
regex 2024.5.15 pypi_0 pypi
requests 2.32.3 pypi_0 pypi
requests-toolbelt 1.0.0 pypi_0 pypi
safetensors 0.4.3 pypi_0 pypi
scikit-learn 1.5.0 pypi_0 pypi
scipy 1.11.4 pypi_0 pypi
sentence-transformers 3.0.1 pypi_0 pypi
sentry-sdk 2.6.0 pypi_0 pypi
setuptools 70.1.0 pyhd8ed1ab_0 conda-forge
six 1.16.0 pyhd3eb1b0_1 anaconda
smmap 5.0.1 pypi_0 pypi
sniffio 1.3.1 pypi_0 pypi
sqlalchemy 2.0.31 pypi_0 pypi
sqlparse 0.5.0 pypi_0 pypi
stack_data 0.2.0 pyhd3eb1b0_0 anaconda
sympy 1.12.1 pypi_0 pypi
tenacity 8.4.1 pypi_0 pypi
threadpoolctl 3.5.0 pypi_0 pypi
tiktoken 0.7.0 pypi_0 pypi
tk 8.6.13 noxft_h4845f30_101 conda-forge
tokenizers 0.19.1 pypi_0 pypi
torch 2.3.1 pypi_0 pypi
tornado 6.4.1 pypi_0 pypi
tqdm 4.66.4 pypi_0 pypi
traitlets 5.7.1 py39h06a4308_0 anaconda
transformers 4.41.2 pypi_0 pypi
triton 2.3.1 pypi_0 pypi
typing-extensions 4.12.2 pypi_0 pypi
typing-inspect 0.9.0 pypi_0 pypi
typing_extensions 4.7.1 py39h06a4308_0 anaconda
tzdata 2024.1 pypi_0 pypi
umap-learn 0.5.6 pypi_0 pypi
urllib3 2.2.2 pypi_0 pypi
wcwidth 0.2.5 pyhd3eb1b0_0 anaconda
wheel 0.43.0 pyhd8ed1ab_1 conda-forge
wrapt 1.16.0 pypi_0 pypi
xxhash 3.4.1 pypi_0 pypi
xyzservices 2024.6.0 pypi_0 pypi
xz 5.2.6 h166bdaf_0 conda-forge
yarl 1.9.4 pypi_0 pypi
zeromq 4.3.4 h2531618_0 anaconda
zipp 3.19.2 pypi_0 pypi
zstandard 0.22.0 pypi_0 pypi
Current Behaviour?
NOTE I added a second comment below which gets to the root of the problem and suggests a fix.
When I call
scan_results = giskard.scan(model=giskard_model)
It uses OpenAI even though the model is configured to call LM Studio
```text
2024-06-20 17:56:49,828 pid:3414141 MainThread httpx INFO HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 429 Too Many Requests"
2024-06-20 17:56:49,829 pid:3414141 MainThread openai._base_client INFO Retrying request to /chat/completions in 0.823091 seconds
[cut here for space]
RateLimitError: Error code: 429 - {'error': {'message': 'You exceeded your current quota, please check your plan and billing details. For more information on this error, read the docs: https://platform.openai.com/docs/guides/error-codes/api-errors.', 'type': 'insufficient_quota', 'param': None, 'code': 'insufficient_quota'}}
I givwe the full response below, but this sticks out:
https://api.openai.com/v1/chat/completions
I have not configured this anywhere to call to openai, rather the I did configure LM Studio.
Is there a way to globally configure Giskard to usin LM Studio?
Standalone code OR list down the steps to reproduce the issue
I'm using LM Studio Model : TheBloke/Llama 2 13B Q 8.0 GGUF Embeddings : nomic-embeded-text
Here is the code I use to get to this point. I'm using Jyputer Notebook so I'll break it out code / response
import requests
import numpy as np
import faiss
from openai import OpenAI
from langchain import PromptTemplate
from langchain.chains import RetrievalQA
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain.vectorstores import FAISS
from langchain.docstore.in_memory import InMemoryDocstore
from langchain.schema import Document
from langchain_openai import ChatOpenAI
import giskard
import pandas as pd
# Initialize OpenAI client for LM Studio embeddings
embedding_client = OpenAI(base_url="http://localhost:5000/v1", api_key="lm-studio")
# Function to get embeddings from LM Studio
def get_embedding(text, model="model-identifier", retries=3, timeout=120):
text = text.replace("\n", " ")
data = {
"input": [text],
"model": model
}
for attempt in range(retries):
try:
response = requests.post("http://localhost:5000/v1/embeddings", json=data, timeout=timeout)
response.raise_for_status() # Raise an error for bad status codes
response_data = response.json()
if 'data' in response_data and len(response_data['data']) > 0:
return response_data['data'][0]['embedding']
return None
except requests.exceptions.RequestException as e:
if attempt < retries - 1:
time.sleep(2 ** attempt) # Exponential backoff
except Exception as e:
if attempt < retries - 1:
time.sleep(2 ** attempt) # Exponential backoff
return None
# Function to load and split PDF using PyMuPDF
def load_pdf(file_path):
import fitz # PyMuPDF
doc = fitz.open(file_path)
text = ""
for page_num in range(len(doc)):
page = doc.load_page(page_num)
text += page.get_text("text")
return text
# Load the PDF
pdf_text = load_pdf("IPCC_AR6_SYR_LongerReport.pdf")
# Split the text into chunks
text_splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=100, add_start_index=True)
texts = text_splitter.split_text(pdf_text)
# Get embeddings for each text chunk
embeddings = [get_embedding(text) for text in texts if get_embedding(text)]
# Convert embeddings to a NumPy array
embeddings = np.array(embeddings, dtype=np.float32)
# Ensure the embeddings have the desired dimension
desired_dimension = 768
if embeddings.shape[1] != desired_dimension:
print(f"Warning: The embedding dimension is {embeddings.shape[1]}, not {desired_dimension}")
# Create a FAISS index
index = faiss.IndexFlatL2(embeddings.shape[1])
index.add(embeddings)
# Prepare documents and docstore
documents = [Document(page_content=text) for text in texts]
docstore = InMemoryDocstore({str(i): doc for i, doc in enumerate(documents)})
# Create an index-to-docstore-id mapping
index_to_docstore_id = {i: str(i) for i in range(len(documents))}
# Use Langchain's FAISS retriever
vectorstore = FAISS(embedding_function=get_embedding, index=index, docstore=docstore, index_to_docstore_id=index_to_docstore_id)
# Prepare QA chain
PROMPT_TEMPLATE = """You are the Climate Assistant, a helpful AI assistant made by Giskard.
Your task is to answer common questions on climate change.
You will be given a question and relevant excerpts from the IPCC Climate Change Synthesis Report (2023).
Please provide short and clear answers based on the provided context. Be polite and helpful.
Context:
{context}
Question:
{question}
Your answer:
"""
llm = ChatOpenAI(base_url="http://localhost:5000/v1", temperature=0.85, api_key="not_needed")
prompt = PromptTemplate(template=PROMPT_TEMPLATE, input_variables=["question", "context"])
climate_qa_chain = RetrievalQA.from_llm(llm=llm, retriever=vectorstore.as_retriever(), prompt=prompt)
response
2024-06-20 17:56:33,100 pid:3414141 MainThread langchain_community.vectorstores.faiss WARNING `embedding_function` is expected to be an Embeddings object, support for passing in a function will soon be removed.
# Example question
question = "What are the main impacts of climate change?"
answer = climate_qa_chain({"query": question})
print(answer['result'])
response
Hi there! As the Climate Assistant, I can tell you that the main impacts of climate change include:
* Increased frequency and severity of extreme weather events such as heatwaves, droughts, floods, and storms (high confidence)
* Rising sea levels and coastal erosion, with associated impacts on coastal ecosystems and human settlements (high confidence)
* Loss of biodiversity, including changes in the distribution and abundance of species, as well as the loss of ecosystem functions and services (very high confidence)
* Negative impacts on human health, including an increased risk of heat-related illness and the spread of disease vectors (high confidence)
* Economic losses and damages to infrastructure, including impacts on agriculture, forestry, fisheries, energy, and tourism (high confidence)
* Increased risk of water scarcity and competition for resources, with potential impacts on food security and human settlements (medium to high confidence)
Overall, the impacts of climate change are far-reaching and can have significant consequences for both the environment and human societies. It is important to take action to mitigate and adapt to these impacts in order to minimize their effects.
def model_predict(df: pd.DataFrame):
"""Wraps the LLM call in a simple Python function.
The function takes a pandas.DataFrame containing the input variables needed
by your model, and must return a list of the outputs (one for each row).
"""
return [climate_qa_chain.run({"query": question}) for question in df["question"]]
# Don’t forget to fill the `name` and `description`: they are used by Giskard
# to generate domain-specific tests.
giskard_model = giskard.Model(
model=model_predict,
model_type="text_generation",
name="Climate Change Question Answering",
description="This model answers any question about climate change based on IPCC reports",
feature_names=["question"],
)
response
2024-06-20 17:56:46,379 pid:3414141 MainThread giskard.models.automodel INFO Your 'prediction_function' is successfully wrapped by Giskard's 'PredictionFunctionModel' wrapper class.
then
scan_results = giskard.scan(model=giskard_model)
full results
2024-06-20 17:56:49,828 pid:3414141 MainThread httpx INFO HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 429 Too Many Requests"
2024-06-20 17:56:49,829 pid:3414141 MainThread openai._base_client INFO Retrying request to /chat/completions in 0.823091 seconds
2024-06-20 17:56:50,729 pid:3414141 MainThread httpx INFO HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 429 Too Many Requests"
2024-06-20 17:56:50,729 pid:3414141 MainThread openai._base_client INFO Retrying request to /chat/completions in 1.627491 seconds
2024-06-20 17:56:52,434 pid:3414141 MainThread httpx INFO HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 429 Too Many Requests"
---------------------------------------------------------------------------
RateLimitError Traceback (most recent call last)
Cell In[7], line 1
----> 1 scan_results = giskard.scan(model=giskard_model)
File /home/michael/anaconda3/envs/giskard/lib/python3.9/site-packages/giskard/scanner/__init__.py:64, in scan(model, dataset, features, params, only, verbose, raise_exceptions)
35 """Automatically detects model vulnerabilities.
36
37 See :class:`Scanner` for more details.
(...)
61 A scan report object containing the results of the scan.
62 """
63 scanner = Scanner(params, only=only)
---> 64 return scanner.analyze(
65 model, dataset=dataset, features=features, verbose=verbose, raise_exceptions=raise_exceptions
66 )
File /home/michael/anaconda3/envs/giskard/lib/python3.9/site-packages/giskard/scanner/scanner.py:100, in Scanner.analyze(self, model, dataset, features, verbose, raise_exceptions)
77 """Runs the analysis of a model and dataset, detecting issues.
78
79 Parameters
(...)
96 A report object containing the detected issues and other information.
97 """
98 with TemporaryRootLogLevel(logging.INFO if verbose else logging.NOTSET):
99 # Check that the model and dataset were appropriately wrapped with Giskard
--> 100 model, dataset, model_validation_time = self._validate_model_and_dataset(model, dataset)
102 # Check that provided features are valid
103 features = self._validate_features(features, model, dataset)
File /home/michael/anaconda3/envs/giskard/lib/python3.9/site-packages/giskard/scanner/scanner.py:210, in Scanner._validate_model_and_dataset(self, model, dataset)
204 if dataset is not None and not isinstance(dataset, Dataset):
205 raise ValueError(
206 "The dataset object you provided is not valid. Please wrap your dataframe with `giskard.Dataset`. "
207 "You can follow the docs here: https://docs.giskard.ai/en/stable/guides/wrap_dataset/index.html"
208 )
--> 210 model, dataset = self._prepare_model_dataset(model, dataset)
212 if not model.is_text_generation:
213 time_start = perf_counter()
File /home/michael/anaconda3/envs/giskard/lib/python3.9/site-packages/giskard/scanner/scanner.py:269, in Scanner._prepare_model_dataset(self, model, dataset)
267 logger.debug("Automatically generating test dataset.")
268 try:
--> 269 return model, generate_test_dataset(model)
270 except LLMGenerationError:
271 warning(
272 "Failed to generate test dataset. Trying to run the scan with an empty dataset. For improved results, please provide a test dataset."
273 )
File /home/michael/anaconda3/envs/giskard/lib/python3.9/site-packages/giskard/llm/utils.py:45, in generate_test_dataset(model, num_samples, temperature, column_types, llm_seed)
14 """Generates a synthetic test dataset using an LLM.
15
16 Parameters
(...)
41 :class:`giskard.llm.generators.BaseDataGenerator`
42 """
43 generator = SimpleDataGenerator(llm_temperature=temperature, llm_seed=llm_seed)
---> 45 return generator.generate_dataset(model, num_samples, column_types)
File /home/michael/anaconda3/envs/giskard/lib/python3.9/site-packages/giskard/llm/generators/base.py:64, in _BaseLLMGenerator.generate_dataset(self, model, num_samples, column_types)
41 """Generates a test dataset for the model.
42
43 Parameters
(...)
60 If the generation fails.
61 """
62 messages = self._format_messages(model, num_samples, column_types)
---> 64 out = self.llm_client.complete(
65 messages=messages,
66 temperature=self.llm_temperature,
67 caller_id=self.__class__.__name__,
68 seed=self.llm_seed,
69 format="json",
70 )
72 generated = self._parse_output(out)
74 dataset = Dataset(
75 df=pd.DataFrame(generated),
76 name=self._make_dataset_name(model),
77 validation=False,
78 column_types=column_types,
79 )
File /home/michael/anaconda3/envs/giskard/lib/python3.9/site-packages/giskard/llm/client/openai.py:63, in OpenAIClient.complete(self, messages, temperature, max_tokens, caller_id, seed, format)
60 extra_params["response_format"] = {"type": "json_object"}
62 try:
---> 63 completion = self._client.chat.completions.create(
64 model=self.model,
65 messages=[asdict(m) for m in messages],
66 temperature=temperature,
67 max_tokens=max_tokens,
68 **extra_params,
69 )
70 except openai.AuthenticationError as err:
71 raise LLMConfigurationError(AUTH_ERROR_MESSAGE) from err
File /home/michael/anaconda3/envs/giskard/lib/python3.9/site-packages/sentry_sdk/utils.py:1711, in ensure_integration_enabled.<locals>.patcher.<locals>.runner(*args, **kwargs)
1708 if sentry_sdk.get_client().get_integration(integration) is None:
1709 return original_function(*args, **kwargs)
-> 1711 return sentry_patched_function(*args, **kwargs)
File /home/michael/anaconda3/envs/giskard/lib/python3.9/site-packages/sentry_sdk/integrations/openai.py:153, in _wrap_chat_completion_create.<locals>.new_chat_completion(*args, **kwargs)
151 _capture_exception(e)
152 span.__exit__(None, None, None)
--> 153 raise e from None
155 integration = sentry_sdk.get_client().get_integration(OpenAIIntegration)
157 with capture_internal_exceptions():
File /home/michael/anaconda3/envs/giskard/lib/python3.9/site-packages/sentry_sdk/integrations/openai.py:149, in _wrap_chat_completion_create.<locals>.new_chat_completion(*args, **kwargs)
147 span.__enter__()
148 try:
--> 149 res = f(*args, **kwargs)
150 except Exception as e:
151 _capture_exception(e)
File /home/michael/anaconda3/envs/giskard/lib/python3.9/site-packages/openai/_utils/_utils.py:277, in required_args.<locals>.inner.<locals>.wrapper(*args, **kwargs)
275 msg = f"Missing required argument: {quote(missing[0])}"
276 raise TypeError(msg)
--> 277 return func(*args, **kwargs)
File /home/michael/anaconda3/envs/giskard/lib/python3.9/site-packages/openai/resources/chat/completions.py:640, in Completions.create(self, messages, model, frequency_penalty, function_call, functions, logit_bias, logprobs, max_tokens, n, parallel_tool_calls, presence_penalty, response_format, seed, service_tier, stop, stream, stream_options, temperature, tool_choice, tools, top_logprobs, top_p, user, extra_headers, extra_query, extra_body, timeout)
606 @required_args(["messages", "model"], ["messages", "model", "stream"])
607 def create(
608 self,
(...)
638 timeout: float | httpx.Timeout | None | NotGiven = NOT_GIVEN,
639 ) -> ChatCompletion | Stream[ChatCompletionChunk]:
--> 640 return self._post(
641 "/chat/completions",
642 body=maybe_transform(
643 {
644 "messages": messages,
645 "model": model,
646 "frequency_penalty": frequency_penalty,
647 "function_call": function_call,
648 "functions": functions,
649 "logit_bias": logit_bias,
650 "logprobs": logprobs,
651 "max_tokens": max_tokens,
652 "n": n,
653 "parallel_tool_calls": parallel_tool_calls,
654 "presence_penalty": presence_penalty,
655 "response_format": response_format,
656 "seed": seed,
657 "service_tier": service_tier,
658 "stop": stop,
659 "stream": stream,
660 "stream_options": stream_options,
661 "temperature": temperature,
662 "tool_choice": tool_choice,
663 "tools": tools,
664 "top_logprobs": top_logprobs,
665 "top_p": top_p,
666 "user": user,
667 },
668 completion_create_params.CompletionCreateParams,
669 ),
670 options=make_request_options(
671 extra_headers=extra_headers, extra_query=extra_query, extra_body=extra_body, timeout=timeout
672 ),
673 cast_to=ChatCompletion,
674 stream=stream or False,
675 stream_cls=Stream[ChatCompletionChunk],
676 )
File /home/michael/anaconda3/envs/giskard/lib/python3.9/site-packages/openai/_base_client.py:1250, in SyncAPIClient.post(self, path, cast_to, body, options, files, stream, stream_cls)
1236 def post(
1237 self,
1238 path: str,
(...)
1245 stream_cls: type[_StreamT] | None = None,
1246 ) -> ResponseT | _StreamT:
1247 opts = FinalRequestOptions.construct(
1248 method="post", url=path, json_data=body, files=to_httpx_files(files), **options
1249 )
-> 1250 return cast(ResponseT, self.request(cast_to, opts, stream=stream, stream_cls=stream_cls))
File /home/michael/anaconda3/envs/giskard/lib/python3.9/site-packages/openai/_base_client.py:931, in SyncAPIClient.request(self, cast_to, options, remaining_retries, stream, stream_cls)
922 def request(
923 self,
924 cast_to: Type[ResponseT],
(...)
929 stream_cls: type[_StreamT] | None = None,
930 ) -> ResponseT | _StreamT:
--> 931 return self._request(
932 cast_to=cast_to,
933 options=options,
934 stream=stream,
935 stream_cls=stream_cls,
936 remaining_retries=remaining_retries,
937 )
File /home/michael/anaconda3/envs/giskard/lib/python3.9/site-packages/openai/_base_client.py:1015, in SyncAPIClient._request(self, cast_to, options, remaining_retries, stream, stream_cls)
1013 if retries > 0 and self._should_retry(err.response):
1014 err.response.close()
-> 1015 return self._retry_request(
1016 options,
1017 cast_to,
1018 retries,
1019 err.response.headers,
1020 stream=stream,
1021 stream_cls=stream_cls,
1022 )
1024 # If the response is streamed then we need to explicitly read the response
1025 # to completion before attempting to access the response text.
1026 if not err.response.is_closed:
File /home/michael/anaconda3/envs/giskard/lib/python3.9/site-packages/openai/_base_client.py:1063, in SyncAPIClient._retry_request(self, options, cast_to, remaining_retries, response_headers, stream, stream_cls)
1059 # In a synchronous context we are blocking the entire thread. Up to the library user to run the client in a
1060 # different thread if necessary.
1061 time.sleep(timeout)
-> 1063 return self._request(
1064 options=options,
1065 cast_to=cast_to,
1066 remaining_retries=remaining,
1067 stream=stream,
1068 stream_cls=stream_cls,
1069 )
File /home/michael/anaconda3/envs/giskard/lib/python3.9/site-packages/openai/_base_client.py:1015, in SyncAPIClient._request(self, cast_to, options, remaining_retries, stream, stream_cls)
1013 if retries > 0 and self._should_retry(err.response):
1014 err.response.close()
-> 1015 return self._retry_request(
1016 options,
1017 cast_to,
1018 retries,
1019 err.response.headers,
1020 stream=stream,
1021 stream_cls=stream_cls,
1022 )
1024 # If the response is streamed then we need to explicitly read the response
1025 # to completion before attempting to access the response text.
1026 if not err.response.is_closed:
File /home/michael/anaconda3/envs/giskard/lib/python3.9/site-packages/openai/_base_client.py:1063, in SyncAPIClient._retry_request(self, options, cast_to, remaining_retries, response_headers, stream, stream_cls)
1059 # In a synchronous context we are blocking the entire thread. Up to the library user to run the client in a
1060 # different thread if necessary.
1061 time.sleep(timeout)
-> 1063 return self._request(
1064 options=options,
1065 cast_to=cast_to,
1066 remaining_retries=remaining,
1067 stream=stream,
1068 stream_cls=stream_cls,
1069 )
File /home/michael/anaconda3/envs/giskard/lib/python3.9/site-packages/openai/_base_client.py:1030, in SyncAPIClient._request(self, cast_to, options, remaining_retries, stream, stream_cls)
1027 err.response.read()
1029 log.debug("Re-raising status error")
-> 1030 raise self._make_status_error_from_response(err.response) from None
1032 return self._process_response(
1033 cast_to=cast_to,
1034 options=options,
(...)
1037 stream_cls=stream_cls,
1038 )
RateLimitError: Error code: 429 - {'error': {'message': 'You exceeded your current quota, please check your plan and billing details. For more information on this error, read the docs: https://platform.openai.com/docs/guides/error-codes/api-errors.', 'type': 'insufficient_quota', 'param': None, 'code': 'insufficient_quota'}}
### Relevant log output
_No response_