Sotiris Anagnostidis

Nono its fine, we can just add it to the dataset __init__ or just the README to be easy to follow the instructions to recreate it

Not exactly what we had in mind, can you check again #1661?

The reward training uses a different dataset to begin with, see [here](https://github.com/LAION-AI/Open-Assistant/blob/main/model/reward/instructor/rank_datasets.py#L298). Then we need to make sure the splits between these datasets are consistent, in case we want to...

If we want to test knowledge/retrieval etc, there are some good evaluations on things like 'wizard of wikipedia' or 'wizard of the internet. If we want to test conversational skills...

So as far as I know :)

yes, please, If anyone is interested I can assign you to it

Hey the code of @shahules786 seems to be taking care of the second part of the evaluation, that is great. Regarding the first point, i.e evaluating on some known datasets...

@theblackcat102 @dvruette if everything looks good we can merge

Hey thanks, seems ray's location has been changed. Made a [PR](https://github.com/CarperAI/trlx/pull/456). I will get back to you. In the meantime you can install ray manually first.