Percy Li

[Suggestion] Consider enhancing the Evaluate class to support callable objects as metrics

1

Can we change the line of code in the Evaluate class below: ``` # Rename the 'correct' column to the name of the metric function #metric_name = metric.__name__ assert(callable(metric)) metric_name...

After saving and loading, the program optimized by BayesianSignatureOptimizer encounters an error when calling forward

5

code as follows: ``` def train(): teleprompter = BayesianSignatureOptimizer(prompt_model=sql_llm, task_model=sql_llm, metric=assess_api_selector, n=10, init_temperature=0.9, verbose=True,track_stats=True) evaluate_api_selector = Evaluate(devset=dataset_api_calling_gt_sample[10:20], metric=assess_api_selector, num_threads=1, display_progress=True, display_table=0) kwargs = dict(num_threads=1, display_progress=True, display_table=0) uncompiled_prompt = APISelector() #evaluate_api_selector(uncompiled_prompt)...

Minor changes to enable the evaluator to accept any callable objects as metrics.