Best Practice for Including Reward-Bench with Local Modifications in Our Repo

[hank0316]

Hi Nathan,

I’m currently preparing to release a new repository that contains the code used in my paper. As part of our experiments, we made some slight modifications to the reward-bench code (we're using the v0.1.0-dev version).

The changes include:

1. Updating the evaluation results to Weights & Biases (wandb).
2. Loading our model using float16, without load_in_8bit=True (since I'm not sure what load_in_8bit actually does, and I just want to load our RM in fp16).
3. Adding a system prompt to the message object passed to tokenizer.apply_chat_template.

I'm reaching out to ask about the best practice for including reward-bench with our changes into our repo. At the moment, we’ve removed the .git directory and committed the entire reward-bench codebase to our repository.

Is there a better approach to incorporate reward-bench while maintaining our local modifications? Any advice would be much appreciated!

Thank you for your time and help!

Best regards, Tzu-Han