dspy icon indicating copy to clipboard operation
dspy copied to clipboard

Avoid magic names

Open thomasnormal opened this issue 1 year ago • 0 comments

Some of the code in template_v2 and v3 gives special treatment to names like context, answer, answers and passages. This is a bit confusing to new users, when they rename their signature, e.g. answers to solutions and suddenly the code stops working.

A way to avoid this, at least in the metric functions, could be using nested functions like this:

def exact_match(attr):
    def exact_match(example, pred, trace=None, frac=1.0):
        example_answer = getattr(example, attr, None)
        assert example_answer is not None
        pred_answer = getattr(pred, attr, None)
        assert pred_answer is not None

        assert(type(example_answer) is str or type(example_answer) is list)
        if type(example_answer) is str:
            return dsp.answer_match(pred_answer, [example_answer], frac=frac)
        else: # type(example_answer) is list
            return dsp.answer_match(pred_answer, example_answer, frac=frac)
    return exact_match

Then you can write metric=exact_match("answer") to get the same effect as the current answer_exact_match function, but you can also insert some other attribute name, if your signatures use some other labels.

I'm not sure how to fix the problem in the case of templates. So I'll just leave this open for discussion.

thomasnormal avatar Feb 05 '24 09:02 thomasnormal