giskard
giskard copied to clipboard
Wrapping custom application endpoint not working
Issue Type
Bug
Source
source
Giskard Library Version
2.11.0
Giskard Hub Version
N/A
OS Platform and Distribution
No response
Python version
3.11.4
Installed python packages
No response
Current Behaviour?
Testing the custom application endpoint that is using AzureOpenAI in the backend is failing using the code snippet provided by giskard to wrap the custom endpoint.
The application endpoint does not require any API key at this time and the giskard.scan requires to pass an OPENAI_API_KEY. If a dummy key is set as environment variable, the code fails with API connection error.
Standalone code OR list down the steps to reproduce the issue
import os
import giskard
from giskard.llm import set_llm_model
def call_my_api():
return [requests.post('https://api-url.com/v1/chatbot/request', json=json_data, headers='Content-type': 'application/json'})]
set_llm_model("gpt4")
# Create a giskard.Model object. Don’t forget to fill the `name` and `description`
giskard_model = giskard.Model(
call_my_api, # our langchain.LLMChain object
model_type="text_generation",
name="My Generic Assistant",
description="A generic assistant that kindly answers questions.",
feature_names=["messages"],
)
scan_results = giskard.scan(giskard_model)
display(scan_results) # in your notebook
Relevant log output
Error:
---------------------------------------------------------------------------
APIConnectionError Traceback (most recent call last)
Cell In[22], line 1
----> 1 scan_results = giskard.scan(giskard_model)
2 display(scan_results) # in your notebook
File /opt/homebrew/lib/python3.11/site-packages/giskard/scanner/__init__.py:64, in scan(model, dataset, features, params, only, verbose, raise_exceptions)
35 """Automatically detects model vulnerabilities.
36
37 See :class:`Scanner` for more details.
(...)
61 A scan report object containing the results of the scan.
62 """
63 scanner = Scanner(params, only=only)
---> 64 return scanner.analyze(
65 model, dataset=dataset, features=features, verbose=verbose, raise_exceptions=raise_exceptions
66 )
File /opt/homebrew/lib/python3.11/site-packages/giskard/scanner/scanner.py:100, in Scanner.analyze(self, model, dataset, features, verbose, raise_exceptions)
77 """Runs the analysis of a model and dataset, detecting issues.
78
79 Parameters
(...)
96 A report object containing the detected issues and other information.
97 """
...
988 'HTTP Request: %s %s "%i %s"', request.method, request.url, response.status_code, response.reason_phrase
989 )
991 try:
APIConnectionError: Connection error.