Azure
[DRAFT][FIX] Allow AAD Auth for AzureContentFilterScorer
Description
The AzureContentFilterScorer required an API key for use, which does not conform to all security policies.
Tests and Documentation
Would welcome information on how to test within your testing framework.
Hey Richard! Thanks for this PR. We don't currently have unit testing around use_aad_auth but you will see that some notebooks will have examples that expose the parameter (such as https://github.com/Azure/PyRIT/blob/11a64f7a34d1f957bd20b9cd1cf5ef46290961d0/doc/code/targets/2_gpt_4o_target.py#L49).
I'd recommend adding an example in this doc: https://github.com/Azure/PyRIT/blob/main/doc/code/scoring/2_azure_content_safety_scorers.py#L33
...where you show how to use AzureContentFilterScorer with the use_aad_auth option. You can set this to False when actually submitting it, but test locally with it set to True to make sure its functioning as expected.
Just trying to update the notebook, I'm getting an error when running:
Error fetching data from table PromptMemoryEntries: (duckdb.duckdb.BinderException) Binder Error: Table "PromptMemoryEntries" does not have a column named "original_prompt_id"
LINE 1: ...MemoryEntries_converted_value_sha256", "PromptMemoryEntries".original_prompt_i...
^
[SQL: SELECT "PromptMemoryEntries".id AS "PromptMemoryEntries_id", "PromptMemoryEntries".role AS "PromptMemoryEntries_role", "PromptMemoryEntries".conversation_id AS "PromptMemoryEntries_conversation_id", "PromptMemoryEntries".sequence AS "PromptMemoryEntries_sequence", "PromptMemoryEntries".timestamp AS "PromptMemoryEntries_timestamp", "PromptMemoryEntries".labels AS "PromptMemoryEntries_labels", "PromptMemoryEntries".prompt_metadata AS "PromptMemoryEntries_prompt_metadata", "PromptMemoryEntries".converter_identifiers AS "PromptMemoryEntries_converter_identifiers", "PromptMemoryEntries".prompt_target_identifier AS "PromptMemoryEntries_prompt_target_identifier", "PromptMemoryEntries".orchestrator_identifier AS "PromptMemoryEntries_orchestrator_identifier", "PromptMemoryEntries".response_error AS "PromptMemoryEntries_response_error", "PromptMemoryEntries".original_value_data_type AS "PromptMemoryEntries_original_value_data_type", "PromptMemoryEntries".original_value AS "PromptMemoryEntries_original_value", "PromptMemoryEntries".original_value_sha256 AS "PromptMemoryEntries_original_value_sha256", "PromptMemoryEntries".converted_value_data_type AS "PromptMemoryEntries_converted_value_data_type", "PromptMemoryEntries".converted_value AS "PromptMemoryEntries_converted_value", "PromptMemoryEntries".converted_value_sha256 AS "PromptMemoryEntries_converted_value_sha256", "PromptMemoryEntries".original_prompt_id AS "PromptMemoryEntries_original_prompt_id"
FROM "PromptMemoryEntries"
WHERE "PromptMemoryEntries".id IN ($1::UUID)]
[parameters: ('3fdb7e3f-2158-4dff-b497-a241a1ae9ce7',)]
(Background on this error at: https://sqlalche.me/e/20/f405)
Traceback (most recent call last):
File "C:\Users\riedgar\AppData\Local\miniconda3\envs\pyrit-310\lib\site-packages\sqlalchemy\engine\base.py", line 1970, in _exec_single_context
self.dialect.do_execute(
File "C:\Users\riedgar\AppData\Local\miniconda3\envs\pyrit-310\lib\site-packages\sqlalchemy\engine\default.py", line 924, in do_execute
cursor.execute(statement, parameters)
File "C:\Users\riedgar\AppData\Local\miniconda3\envs\pyrit-310\lib\site-packages\duckdb_engine\__init__.py", line 162, in execute
self.__c.execute(statement, parameters)
duckdb.duckdb.BinderException: Binder Error: Table "PromptMemoryEntries" does not have a column named "original_prompt_id"
LINE 1: ...MemoryEntries_converted_value_sha256", "PromptMemoryEntries".original_prompt_i...
^
I don't think that's my fault.... at least due to the changes I've made.
Just trying to update the notebook, I'm getting an error when running:
Error fetching data from table PromptMemoryEntries: (duckdb.duckdb.BinderException) Binder Error: Table "PromptMemoryEntries" does not have a column named "original_prompt_id" LINE 1: ...MemoryEntries_converted_value_sha256", "PromptMemoryEntries".original_prompt_i... ^ [SQL: SELECT "PromptMemoryEntries".id AS "PromptMemoryEntries_id", "PromptMemoryEntries".role AS "PromptMemoryEntries_role", "PromptMemoryEntries".conversation_id AS "PromptMemoryEntries_conversation_id", "PromptMemoryEntries".sequence AS "PromptMemoryEntries_sequence", "PromptMemoryEntries".timestamp AS "PromptMemoryEntries_timestamp", "PromptMemoryEntries".labels AS "PromptMemoryEntries_labels", "PromptMemoryEntries".prompt_metadata AS "PromptMemoryEntries_prompt_metadata", "PromptMemoryEntries".converter_identifiers AS "PromptMemoryEntries_converter_identifiers", "PromptMemoryEntries".prompt_target_identifier AS "PromptMemoryEntries_prompt_target_identifier", "PromptMemoryEntries".orchestrator_identifier AS "PromptMemoryEntries_orchestrator_identifier", "PromptMemoryEntries".response_error AS "PromptMemoryEntries_response_error", "PromptMemoryEntries".original_value_data_type AS "PromptMemoryEntries_original_value_data_type", "PromptMemoryEntries".original_value AS "PromptMemoryEntries_original_value", "PromptMemoryEntries".original_value_sha256 AS "PromptMemoryEntries_original_value_sha256", "PromptMemoryEntries".converted_value_data_type AS "PromptMemoryEntries_converted_value_data_type", "PromptMemoryEntries".converted_value AS "PromptMemoryEntries_converted_value", "PromptMemoryEntries".converted_value_sha256 AS "PromptMemoryEntries_converted_value_sha256", "PromptMemoryEntries".original_prompt_id AS "PromptMemoryEntries_original_prompt_id" FROM "PromptMemoryEntries" WHERE "PromptMemoryEntries".id IN ($1::UUID)] [parameters: ('3fdb7e3f-2158-4dff-b497-a241a1ae9ce7',)] (Background on this error at: https://sqlalche.me/e/20/f405) Traceback (most recent call last): File "C:\Users\riedgar\AppData\Local\miniconda3\envs\pyrit-310\lib\site-packages\sqlalchemy\engine\base.py", line 1970, in _exec_single_context self.dialect.do_execute( File "C:\Users\riedgar\AppData\Local\miniconda3\envs\pyrit-310\lib\site-packages\sqlalchemy\engine\default.py", line 924, in do_execute cursor.execute(statement, parameters) File "C:\Users\riedgar\AppData\Local\miniconda3\envs\pyrit-310\lib\site-packages\duckdb_engine\__init__.py", line 162, in execute self.__c.execute(statement, parameters) duckdb.duckdb.BinderException: Binder Error: Table "PromptMemoryEntries" does not have a column named "original_prompt_id" LINE 1: ...MemoryEntries_converted_value_sha256", "PromptMemoryEntries".original_prompt_i... ^I don't think that's my fault.... at least due to the changes I've made.
Did you try deleting the local DuckDB in the results folder? Might be related to recent schema changes...
