Azure
BUG get_scores_by_memory_labels and a few other memory methods do not scale
Describe the bug
A few methods in the memory interface don't scale beyond a few records. This is because they query by ID and have to insert all the IDs into the query which runs into a limit.
What should be done about it
We should
a) find out what the limit is (per memory type, but fine to start with DuckDB only)
b) edit get_scores_by_prompt_ids to batch the requests, i.e., if you're given 10000 IDs then don't put all of them into a single request, but rather send them in batches of 100 (or whatever the limit from a) is).
The affected methods as far as I can tell are all the ones using get_scores_by_prompt_ids
We also need unit tests for these methods to do the batching properly, and integration tests where we exceed the limit and it still works.
Steps/Code to Reproduce
Just call any of these methods with 1000 scores and it'll fail.
Expected Results
Should retrieve the scores without error.
Actual Results
exception
Hi, is this open for assignment? I'm relatively new to PyRIT and was thinking it might be good to explore and understand the project as well!
It is! Reach out if you have questions. This should be doable with a local DuckDB provided you add a few hundred entries.