dspy.Assert/Suggest

[fdzr]

Suppose I have the following program

def custom_evaluate(dataset, metric, model, debug=False):
    acc = 0
    cont = 1

    for item in dataset:
        pred = model(
            sentence1=item.sentence1,
            sentence2=item.sentence2,
            target_word=item.target_word,
        )

        if pred.answer == item.answer:
            acc += 1

        if debug is True:
            print("Prediction: ", pred.answer)

    print(f"Accuracy: {acc * 100 / len(dataset)}")

and my model or module has a dspy.Suggest instruction. I would like to know how to process this type of output 2024/11/28 15:33:48 INFO dspy.primitives.assertions: SuggestionFailed:

I mean, I want to know how to calculate accuracy over the examples with right format, skip or count the examples with bad format.