Open-Assistant icon indicating copy to clipboard operation
Open-Assistant copied to clipboard

Evaluation of HC3 dataset against our models

Open huu4ontocord opened this issue 2 years ago • 4 comments

We need to create eval code to run against the various questions from https://huggingface.co/datasets/Hello-SimpleAI/HC3 at least a subset that we won't tain on.

And we need to eval against the output.

huu4ontocord avatar Jan 23 '23 00:01 huu4ontocord

see also https://huggingface.co/datasets/xzyao/HC3_Galactica

huu4ontocord avatar Jan 23 '23 01:01 huu4ontocord

Also run against RWKV bot as baseline: https://colab.research.google.com/github/harrisonvanderbyl/rwkv_chatbot/blob/main/Example.ipynb#scrollTo=u4xh9ClVQ1-g with this prompt: f'\nQ & A\n\nQuestion:\n{qq}\n\nDetailed Expert Answer:\n{aa}' (let the model generate after "Answer:\n"

huu4ontocord avatar Jan 23 '23 01:01 huu4ontocord

Watching this.

sbmaruf avatar Jan 24 '23 04:01 sbmaruf

cc @theblackcat102

huu4ontocord avatar Jan 25 '23 23:01 huu4ontocord