Azure
FEAT: Add Unify Integration for Multi-Provider LLM Support
Is your feature request related to a problem? Please describe.
PyRIT currently lacks built-in support for easily using and comparing multiple LLM providers. This makes it challenging for users to leverage and evaluate different models efficiently within the PyRIT framework.
Describe the solution you'd like
Integrate Unify's functionality into PyRIT to enable:
Universal API access to multiple LLM providers through a single interface Built-in benchmarking and evaluation tools for comparing LLM performance across providers Intelligent routing capabilities to select optimal models based on specific tasks, quality, cost, and speed The integration should align with PyRIT's existing architecture and follow best practices for extensibility.
Describe alternatives you've considered, if relevant
Implementing provider-specific integrations individually within PyRIT Using a different multi-provider LLM library instead of Unify Creating a custom solution for multi-provider support within PyRIT
Additional context:
Unify documentation: [link to Unify docs] Key features to consider implementing:
Single interface for accessing multiple LLM providers Custom testing and evaluation framework for benchmarking Routing system based on quality, cost, and speed metrics Integration with PyRIT's existing testing and evaluation capabilities This integration would significantly enhance PyRIT's capabilities for AI red teaming across multiple LLM providers and models.
There are several providers for "all-in-one" platforms, e.g., Poe is another one. It's not obvious to me if there are pros/cons to any of them. Any thoughts?
Are you proposing this as something you want to contribute or as a feature request that you would like to see (but you don't want to contribute yourself)?
Hello @romanlutz , I am proposing as something , we at Unify want to add as a feature . I am part of a team (contributors) tasked with an integration like this.
About Poe, I guess Unify is set apart because it provides all LLM providers with one single API key, also adding features like optimisation, benchmarking , routing etc.
Got it! Thanks for confirming. Need to check with our team and will get back to you on Tuesday (Monday is a holiday in the US).
@KatoStevenMubiru we discussed it and suggest you go ahead with a pull request. You can use any other target as a starting point.
Since we don't have Unify accounts we can't add notebooks at this time (as we would be unable to run them). The PR should have at least basic unit tests.
Is it fair to assume that you'll be willing to maintain the target and investigate issues if any should arise?
This is great @romanlutz .
Yes we are willing to maintain the target and investigate incase of any potential issues.
Best regards👏🏾