Azure
[BREAKING] FEAT: Ensemble scoring for Crescendo
Description
This change creates a full pipeline for performing ensemble scoring with crescendo. Included are two new scorers: EnsembleScorer which is the driver of this change and allows results of many scorers to be aggregated, as well as SubstringsMultipleScorer which extends SubstringScorer to allow multiple strings to be searched for in a response. In addition, the crescendo orchestrator has been updated to abstract out the logic for creating the objective scorer. This is now created outside of the orchestrator in a new notebook which has been created as a template to demonstrate the capabilities of a crescendo ensemble orchestrator.
Received support from @eugeniavkim @jbolor21.
This change is breaking because it changes how a CrescendoOrchestrator object is instantiated. Instead of providing a PromptChatTarget as a scoring target for the scorer, the user needs to create a Scorer object outside of the CrescendoOrchestrator and then pass it to objective_float_scale_scorer to be used for scoring. This just abstracts the objective scorer outside of the Orchestrator object and allows for more flexibility.
Tests and Documentation
Still in pogress
I think this has some merge conflicts & pipeline errors! Also would be great if you wrote some docs on the ensemble scorer itself and how this differs from the composite_scorer !
