Open-Assistant
Open-Assistant copied to clipboard
Evaluation of HC3 dataset against our models
We need to create eval code to run against the various questions from https://huggingface.co/datasets/Hello-SimpleAI/HC3 at least a subset that we won't tain on.
And we need to eval against the output.
see also https://huggingface.co/datasets/xzyao/HC3_Galactica
Also run against RWKV bot as baseline: https://colab.research.google.com/github/harrisonvanderbyl/rwkv_chatbot/blob/main/Example.ipynb#scrollTo=u4xh9ClVQ1-g with this prompt: f'\nQ & A\n\nQuestion:\n{qq}\n\nDetailed Expert Answer:\n{aa}' (let the model generate after "Answer:\n"
Watching this.
cc @theblackcat102