yeoedward

Implement adaption prompt from Llama-Adapter paper

12

Issue #235 Paper: https://arxiv.org/abs/2303.16199 Official Implementation: https://github.com/ZrrSkywalker/LLaMA-Adapter This implementation is quite specific to Llama although in principle the same technique can be used for other transformer models. It is implemented...

Pile tasks on big-refactor use dataset_names from old dataset loader that don't exist on HF

2

Task example: https://github.com/EleutherAI/lm-evaluation-harness/blob/big-refactor/lm_eval/tasks/pile/pile_arxiv.yaml#L7 HF dataset: https://huggingface.co/datasets/EleutherAI/pile Original dataset loader prior to `big-refactor`: https://github.com/EleutherAI/lm-evaluation-harness/blob/master/lm_eval/datasets/pile/pile.py @haileyschoelkopf mentioned that using this loading script should work if we upload it to HF and point...

HH scores summed along batch dimension

4

The hh scores seem to be summed along the batch dimension, which is strange as they are sequence-dependent. Shouldn't separate hh scores be maintained for each sequence in a batch?...