qlora
qlora copied to clipboard
Can you help implement MMLU evals into Axolotl?
https://github.com/OpenAccess-AI-Collective/axolotl/pull/441
return self.call_event("on_evaluate", args, state, control, metrics=metrics)
File "/home/teknium/evaltest/axolotl/venv/lib/python3.10/site-packages/transformers/trainer_callback.py", line 406, in call_event
result = getattr(callback, event)(
File "/home/teknium/evaltest/axolotl/src/axolotl/utils/callbacks.py", line 166, in on_evaluate
(loss, logits, labels) = trainer.prediction_step(
File "/home/teknium/evaltest/axolotl/venv/lib/python3.10/site-packages/transformers/trainer.py", line 3329, in prediction_step
has_labels = False if len(self.label_names) == 0 else all(inputs.get(k) is not None for k in self.label_names)
File "/home/teknium/evaltest/axolotl/venv/lib/python3.10/site-packages/transformers/trainer.py", line 3329, in <genexpr>
has_labels = False if len(self.label_names) == 0 else all(inputs.get(k) is not None for k in self.label_names)
AttributeError: 'NoneType' object has no attribute 'get'