qlora icon indicating copy to clipboard operation
qlora copied to clipboard

Can you help implement MMLU evals into Axolotl?

Open teknium1 opened this issue 1 year ago • 0 comments

https://github.com/OpenAccess-AI-Collective/axolotl/pull/441

    return self.call_event("on_evaluate", args, state, control, metrics=metrics)
  File "/home/teknium/evaltest/axolotl/venv/lib/python3.10/site-packages/transformers/trainer_callback.py", line 406, in call_event
    result = getattr(callback, event)(
  File "/home/teknium/evaltest/axolotl/src/axolotl/utils/callbacks.py", line 166, in on_evaluate
    (loss, logits, labels) = trainer.prediction_step(
  File "/home/teknium/evaltest/axolotl/venv/lib/python3.10/site-packages/transformers/trainer.py", line 3329, in prediction_step
    has_labels = False if len(self.label_names) == 0 else all(inputs.get(k) is not None for k in self.label_names)
  File "/home/teknium/evaltest/axolotl/venv/lib/python3.10/site-packages/transformers/trainer.py", line 3329, in <genexpr>
    has_labels = False if len(self.label_names) == 0 else all(inputs.get(k) is not None for k in self.label_names)
AttributeError: 'NoneType' object has no attribute 'get'

teknium1 avatar Aug 20 '23 02:08 teknium1