transfer-learning-conv-ai
transfer-learning-conv-ai copied to clipboard
Error when setting num_candidates=1 and in dataset there's only one candidate during validation
When setting args.num_candidates to 1, and the actual length of the candidates list of each entry is 1, I get this error during validation:
ERROR:ignite.engine.engine.Engine:Current run is terminating due to exception: For binary cases, y_pred must be comprised of 0's and 1's..
ERROR:ignite.engine.engine.Engine:Engine run is terminating due to exception: For binary cases, y_pred must be comprised of 0's and 1's..
ERROR:ignite.engine.engine.Engine:Engine run is terminating due to exception: For binary cases, y_pred must be comprised of 0's and 1's..
Traceback (most recent call last):
File "./train-regular.py", line 277, in <module>
train()
File "./train-regular.py", line 269, in train
trainer.run(train_loader, max_epochs=args.n_epochs)
File "/usr/local/lib/python3.5/dist-packages/ignite/engine/engine.py", line 446, in run
self._handle_exception(e)
File "/usr/local/lib/python3.5/dist-packages/ignite/engine/engine.py", line 410, in _handle_exception
raise e
File "/usr/local/lib/python3.5/dist-packages/ignite/engine/engine.py", line 437, in run
self._fire_event(Events.EPOCH_COMPLETED)
File "/usr/local/lib/python3.5/dist-packages/ignite/engine/engine.py", line 345, in _fire_event
func(self, *(event_args + args), **kwargs)
File "./train-regular.py", line 223, in <lambda>
trainer.add_event_handler(Events.EPOCH_COMPLETED, lambda _: evaluator.run(val_loader))
File "/usr/local/lib/python3.5/dist-packages/ignite/engine/engine.py", line 446, in run
self._handle_exception(e)
File "/usr/local/lib/python3.5/dist-packages/ignite/engine/engine.py", line 410, in _handle_exception
raise e
File "/usr/local/lib/python3.5/dist-packages/ignite/engine/engine.py", line 433, in run
hours, mins, secs = self._run_once_on_dataset()
File "/usr/local/lib/python3.5/dist-packages/ignite/engine/engine.py", line 399, in _run_once_on_dataset
self._handle_exception(e)
File "/usr/local/lib/python3.5/dist-packages/ignite/engine/engine.py", line 410, in _handle_exception
raise e
File "/usr/local/lib/python3.5/dist-packages/ignite/engine/engine.py", line 392, in _run_once_on_dataset
self._fire_event(Events.ITERATION_COMPLETED)
File "/usr/local/lib/python3.5/dist-packages/ignite/engine/engine.py", line 345, in _fire_event
func(self, *(event_args + args), **kwargs)
File "/usr/local/lib/python3.5/dist-packages/torch/autograd/grad_mode.py", line 49, in decorate_no_grad
return func(*args, **kwargs)
File "/usr/local/lib/python3.5/dist-packages/ignite/metrics/metric.py", line 65, in iteration_completed
self.update(output)
File "/usr/local/lib/python3.5/dist-packages/ignite/metrics/accuracy.py", line 126, in update
self._check_type((y_pred, y))
File "/usr/local/lib/python3.5/dist-packages/ignite/metrics/accuracy.py", line 57, in _check_type
self._check_binary_multilabel_cases((y_pred, y))
File "/usr/local/lib/python3.5/dist-packages/ignite/metrics/accuracy.py", line 48, in _check_binary_multilabel_cases
raise ValueError("For binary cases, y_pred must be comprised of 0's and 1's.")
ValueError: For binary cases, y_pred must be comprised of 0's and 1's.
I have the same problem, did you solve it?
I think num_candidates=1 won't work without significant code modification (removing multiple choice head). I think I've said something different in a different thread, which was wrong.
@sshleifer It seems like the problem is not the num_candidates=1. The model sees a binary classification task when there is only one candidate in the utterance sample.
@sshleifer Do you have a workaround if I don't have multiple candidates for my dataset?
It would require significant code modification. I'd start by
- finding all mentions of multiple choice candidates (and removing them, fixing downstream if they are required)
- updating train.py to not use the CrossEntropyLoss/ or log it
there will probably be more bugs after that!
@sshleifer Yeah, probably not worth changing the code that much. I was talking about a hack, like having the multiple-choice candidates be some random string, say "Something.". Would that work?
Sounds like it might to me. Try it!