test-suite-sql-eval icon indicating copy to clipboard operation
test-suite-sql-eval copied to clipboard

Published 20 hours ago • verified publisher icon taoyds

The metric does not give same report

Open PosoSAgapo opened this issue 3 years ago • 0 comments

Thank you for your contribution to give this unified framework to test on different datasets. However, as I noticed in the Sparc dataset, the leader board has 2 metrics. One is the question match and another is the interaction match. However, the evaluation code in this repository seems to only give turn-based exact match and is not able to give the question match and interaction match. Did I miss anything or does the submission code actually use another evaluation code?

PosoSAgapo avatar Feb 21 '22 07:02 PosoSAgapo