sempre icon indicating copy to clipboard operation
sempre copied to clipboard

Published 20 hours ago • verified publisher icon percyliang

Benchmarking sempre and parasempre 1.0

Open ysu1989 opened this issue 9 years ago • 5 comments

Hi,

I'd like to benchmark sempre and parasempre with a new set of ground-truth questions to evaluate the accuracy and runtime of the systems. I'm using sempre 1.0 so that I can reproduce the results in the papers. My question is concerned with the parameters to run the systems, and it consists of two parts:

  1. If I test the systems with pre-trained models (e.g., the model trained on WebQuestions), what's the most appropriate configuration for each system in order for them to achieve their best performance (in both accuracy and runtime).
  2. If I split the new question set into a training set and a test set, what's the best configuration for each system then for both training and testing?

Thanks in advance for taking care of the issue, and feel free to let me know if there is any confusion in the question.

ysu1989 avatar Jun 11 '15 16:06 ysu1989

Do you know where to download sempre 1.0?

RobinCai1993 avatar Oct 27 '16 12:10 RobinCai1993

You can find it as a previous release: https://github.com/percyliang/sempre/releases

ysu1989 avatar Oct 27 '16 20:10 ysu1989

Thank you! And did you work it on mac os or Ubuntu? Because I am using mac and can not get through the 'make install' step of "install database". Could you give me some suggestions?

RobinCai1993 avatar Oct 28 '16 05:10 RobinCai1993

I used Ubuntu. I guess Linux is suggested. Not sure whether it has been tested on Mac.

On Thu, Oct 27, 2016 at 10:04 PM, RobinCai1993 [email protected] wrote:

Thank you! And did you work it on mac os or Ubuntu? Because I am using mac and can not get through the 'make install' step of "install database". Could you give me some suggestions?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/percyliang/sempre/issues/59#issuecomment-256837048, or mute the thread https://github.com/notifications/unsubscribe-auth/AEJdlAtQ342tW1l7o12U3WtTFvD8KD_3ks5q4YJYgaJpZM4E_8JH .

Yu Su

PhD Candidate, Department of Computer Science

University of California, Santa Barbara

CA, 93117. United States.

Tel: (805)886-7320

ysu1989 avatar Oct 28 '16 07:10 ysu1989

Thank you! I've solved it by using "brew install virtuoso" and change the content of virtuoso.init file in the installed path.

RobinCai1993 avatar Oct 28 '16 13:10 RobinCai1993