LawBench icon indicating copy to clipboard operation
LawBench copied to clipboard

Published 20 hours ago • verified publisher icon open-compass

how to calculate the overall "AVG" score of memorization, understanding and application tasks

Open tonywang-sh opened this issue 1 year ago • 1 comments

As shown in LawBench paper, according to the one-shot and zero-shot scores for the memorization, understanding and application in Table 27 and 31, how to calculate the overall "AVG" score? is it the average of the three tasks? but it is not correct from the Table 27 and 31.

image image

tonywang-sh avatar May 19 '24 04:05 tonywang-sh

Hello, I find this is the weighted average. That is, Overall = Mem / 10 + Understanding / 2 + Application / 2.5

SP-FA avatar Aug 01 '24 18:08 SP-FA