giskard
giskard copied to clipboard
Giskard-AI
Implement a Gemini `LLMClient`
🚀 Feature Request
Create a Gemini client that extends LLMClient.
🔈 Motivation
This will allow to run the scan and tests using Gemini models
🛰 Alternatives
N/A
📎 Additional context
Example can be seen with mistral and openai.
Hello, is this implementation still needed ? I would like to contribute to this issue
Hello @marouanetalaa ,
yes we still need this implementation. It would be appreciated if you can contribute to this issue!
I assigned you to it, let me know if you need help or information.
Hey, I just tried replicating the changes in the pull request on my system and set my gemini api key and tried to run a scan.
I get this for all the LLM Assisted Detectors
2024-07-02 22:16:07,754 pid:17976 MainThread giskard.scanner.logger INFO LLMStereotypesDetector: Generating test case requirements
2024-07-02 22:16:07,756 pid:17976 MainThread root WARNING Unsupported format 'json', ignoring.
2024-07-02 22:16:07,758 pid:17976 MainThread giskard.scanner.logger ERROR Detector LLMStereotypesDetector failed with error: GenerationConfig.__init__() got an unexpected keyword argument 'seed'
I have added these lines of code to setup the client
import google.generativeai as genai
from giskard.llm.client.gemini import GeminiClient
genai.configure(api_key=os.environ["GEMINI_API_KEY"])
giskard.llm.set_default_client(GeminiClient())
Hey, I just tried replicating the changes in the pull request on my system and set my gemini api key and tried to run a scan.
I get this for all the LLM Assisted Detectors
2024-07-02 22:16:07,754 pid:17976 MainThread giskard.scanner.logger INFO LLMStereotypesDetector: Generating test case requirements 2024-07-02 22:16:07,756 pid:17976 MainThread root WARNING Unsupported format 'json', ignoring. 2024-07-02 22:16:07,758 pid:17976 MainThread giskard.scanner.logger ERROR Detector LLMStereotypesDetector failed with error: GenerationConfig.__init__() got an unexpected keyword argument 'seed'I have added these lines of code to setup the client
import google.generativeai as genai from giskard.llm.client.gemini import GeminiClient genai.configure(api_key=os.environ["GEMINI_API_KEY"]) giskard.llm.set_default_client(GeminiClient())
Thanks for reporting the issue, I corrected this inside of #1975
Hey, now I'm having this issue
2024-07-03 20:40:01,087 pid:11428 MainThread giskard.scanner.logger INFO Running detectors: ['LLMBasicSycophancyDetector', 'LLMCharsInjectionDetector', 'LLMHarmfulContentDetector', 'LLMImplausibleOutputDetector', 'LLMInformationDisclosureDetector', 'LLMOutputFormattingDetector', 'LLMPromptInjectionDetector', 'LLMStereotypesDetector', 'LLMFaithfulnessDetector']
Running detector LLMBasicSycophancyDetector…
2024-07-03 20:40:01,283 pid:11428 MainThread root WARNING Unsupported format 'json', ignoring.
2024-07-03 20:40:03,589 pid:11428 MainThread giskard.scanner.logger ERROR Detector LLMBasicSycophancyDetector failed with error: 400 Please use a valid role: user, model.
Traceback (most recent call last):
File "e:\Work\1. Projects\giskard_demo\.venv\Lib\site-packages\giskard\scanner\scanner.py", line 152, in _run_detectors
detected_issues = detector.run(model, dataset, features=features)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "e:\Work\1. Projects\giskard_demo\.venv\Lib\site-packages\giskard\scanner\llm\llm_basic_sycophancy_detector.py", line 85, in run
dataset1, dataset2 = generator.generate_dataset(
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "e:\Work\1. Projects\giskard_demo\.venv\Lib\site-packages\giskard\llm\generators\sycophancy.py", line 101, in generate_dataset
out = self.llm_client.complete(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "e:\Work\1. Projects\giskard_demo\.venv\Lib\site-packages\giskard\llm\client\gemini.py", line 75, in complete
contents=_format(messages),
^^^^^^^^^^^^^^^^^^
File "e:\Work\1. Projects\giskard_demo\.venv\Lib\site-packages\google\generativeai\generative_models.py", line 331, in generate_content
response = self._client.generate_content(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "e:\Work\1. Projects\giskard_demo\.venv\Lib\site-packages\google\ai\generativelanguage_v1beta\services\generative_service\client.py", line 827, in generate_content
response = rpc(
^^^^
File "e:\Work\1. Projects\giskard_demo\.venv\Lib\site-packages\google\api_core\gapic_v1\method.py", line 131, in __call__
return wrapped_func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "e:\Work\1. Projects\giskard_demo\.venv\Lib\site-packages\google\api_core\retry\retry_unary.py", line 293, in retry_wrapped_func
return retry_target(
^^^^^^^^^^^^^
File "e:\Work\1. Projects\giskard_demo\.venv\Lib\site-packages\google\api_core\retry\retry_unary.py", line 153, in retry_target
_retry_error_helper(
File "e:\Work\1. Projects\giskard_demo\.venv\Lib\site-packages\google\api_core\retry\retry_base.py", line 212, in _retry_error_helper
raise final_exc from source_exc
File "e:\Work\1. Projects\giskard_demo\.venv\Lib\site-packages\google\api_core\retry\retry_unary.py", line 144, in retry_target
result = target()
^^^^^^^^
File "e:\Work\1. Projects\giskard_demo\.venv\Lib\site-packages\google\api_core\timeout.py", line 120, in func_with_timeout
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "e:\Work\1. Projects\giskard_demo\.venv\Lib\site-packages\google\api_core\grpc_helpers.py", line 78, in error_remapped_callable
raise exceptions.from_grpc_error(exc) from exc
google.api_core.exceptions.InvalidArgument: 400 Please use a valid role: user, model.
LLMBasicSycophancyDetector: 0 issue detected. (Took 0:00:02.711737)
I think Gemini only accepts user or model as role.
Also https://ai.google.dev/gemini-api/docs/system-instructions?lang=python is a thing. How would we ideally use this within the GeminiClient?
Hey, now I'm having this issue
2024-07-03 20:40:01,087 pid:11428 MainThread giskard.scanner.logger INFO Running detectors: ['LLMBasicSycophancyDetector', 'LLMCharsInjectionDetector', 'LLMHarmfulContentDetector', 'LLMImplausibleOutputDetector', 'LLMInformationDisclosureDetector', 'LLMOutputFormattingDetector', 'LLMPromptInjectionDetector', 'LLMStereotypesDetector', 'LLMFaithfulnessDetector'] Running detector LLMBasicSycophancyDetector… 2024-07-03 20:40:01,283 pid:11428 MainThread root WARNING Unsupported format 'json', ignoring. 2024-07-03 20:40:03,589 pid:11428 MainThread giskard.scanner.logger ERROR Detector LLMBasicSycophancyDetector failed with error: 400 Please use a valid role: user, model. Traceback (most recent call last): File "e:\Work\1. Projects\giskard_demo\.venv\Lib\site-packages\giskard\scanner\scanner.py", line 152, in _run_detectors detected_issues = detector.run(model, dataset, features=features) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "e:\Work\1. Projects\giskard_demo\.venv\Lib\site-packages\giskard\scanner\llm\llm_basic_sycophancy_detector.py", line 85, in run dataset1, dataset2 = generator.generate_dataset( ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "e:\Work\1. Projects\giskard_demo\.venv\Lib\site-packages\giskard\llm\generators\sycophancy.py", line 101, in generate_dataset out = self.llm_client.complete( ^^^^^^^^^^^^^^^^^^^^^^^^^ File "e:\Work\1. Projects\giskard_demo\.venv\Lib\site-packages\giskard\llm\client\gemini.py", line 75, in complete contents=_format(messages), ^^^^^^^^^^^^^^^^^^ File "e:\Work\1. Projects\giskard_demo\.venv\Lib\site-packages\google\generativeai\generative_models.py", line 331, in generate_content response = self._client.generate_content( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "e:\Work\1. Projects\giskard_demo\.venv\Lib\site-packages\google\ai\generativelanguage_v1beta\services\generative_service\client.py", line 827, in generate_content response = rpc( ^^^^ File "e:\Work\1. Projects\giskard_demo\.venv\Lib\site-packages\google\api_core\gapic_v1\method.py", line 131, in __call__ return wrapped_func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "e:\Work\1. Projects\giskard_demo\.venv\Lib\site-packages\google\api_core\retry\retry_unary.py", line 293, in retry_wrapped_func return retry_target( ^^^^^^^^^^^^^ File "e:\Work\1. Projects\giskard_demo\.venv\Lib\site-packages\google\api_core\retry\retry_unary.py", line 153, in retry_target _retry_error_helper( File "e:\Work\1. Projects\giskard_demo\.venv\Lib\site-packages\google\api_core\retry\retry_base.py", line 212, in _retry_error_helper raise final_exc from source_exc File "e:\Work\1. Projects\giskard_demo\.venv\Lib\site-packages\google\api_core\retry\retry_unary.py", line 144, in retry_target result = target() ^^^^^^^^ File "e:\Work\1. Projects\giskard_demo\.venv\Lib\site-packages\google\api_core\timeout.py", line 120, in func_with_timeout return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "e:\Work\1. Projects\giskard_demo\.venv\Lib\site-packages\google\api_core\grpc_helpers.py", line 78, in error_remapped_callable raise exceptions.from_grpc_error(exc) from exc google.api_core.exceptions.InvalidArgument: 400 Please use a valid role: user, model. LLMBasicSycophancyDetector: 0 issue detected. (Took 0:00:02.711737)I think Gemini only accepts user or model as role.
Also https://ai.google.dev/gemini-api/docs/system-instructions?lang=python is a thing. How would we ideally use this within the GeminiClient?
You're right, it will be more robust to switch to the system_instruction. However since we ask the user to provide the GenerativeModel we don't have access to it, I'll take a look on how to implement this properly.
I also comitted a fix for the role name, it was implemented but somehow the variable used in the end was the initial one