README vs Defaults: Which training parameters lead to Hits@1 over 79

[starlightwy]

Hi team thank you very much for the great work and the clean code! I got some problem while running the code and was wondering if you could give me some help :-)

1. I just got an error while I'm trying to evaluate the F1 score with convai_evaluation.py:

Traceback (most recent call last): File "../convai_evaluation.py", line 239, in eval_fct(opt) File "/home/wang/PycharmProjects/transfer-learning-conv-ai/ParlAI/projects/convai2/eval_f1.py", line 27, in eval_f1 report = eval_model(opt, print_parser) File "/home/wang/PycharmProjects/transfer-learning-conv-ai/ParlAI/parlai/scripts/eval_model.py", line 84, in eval_model world.parley() File "/home/wang/PycharmProjects/transfer-learning-conv-ai/ParlAI/parlai/core/worlds.py", line 275, in parley acts[1] = agents[1].act() File "/home/wang/PycharmProjects/transfer-learning-conv-ai/convai_evaluation.py", line 156, in act out_ids, _ = sample_sequence(self.persona, self.history, self.tokenizer, self.model_checkpoint, self.args) File "/home/wang/PycharmProjects/transfer-learning-conv-ai/interact.py", line 74, in sample_sequence logits = top_filtering(logits, top_k=args.top_k, top_p=args.top_p) AttributeError: 'AttrDict' object has no attribute 'top_p'

Then I tried to add an top_p argument but then got:

Traceback (most recent call last): File "../convai_evaluation.py", line 239, in eval_fct(opt) File "/home/wang/PycharmProjects/transfer-learning-conv-ai/ParlAI/projects/convai2/eval_f1.py", line 27, in eval_f1 report = eval_model(opt, print_parser) File "/home/wang/PycharmProjects/transfer-learning-conv-ai/ParlAI/parlai/scripts/eval_model.py", line 84, in eval_model world.parley() File "/home/wang/PycharmProjects/transfer-learning-conv-ai/ParlAI/parlai/core/worlds.py", line 275, in parley acts[1] = agents[1].act() File "/home/wang/PycharmProjects/transfer-learning-conv-ai/convai_evaluation.py", line 156, in act out_ids, _ = sample_sequence(self.persona, self.history, self.tokenizer, self.model_checkpoint, self.args) ValueError: too many values to unpack (expected 2)

I changed this line (line 151 in the original script convai_evaluation.py) from out_ids, _ = sample_sequence(self.persona, self.history, self.tokenizer, self.model_checkpoint, self.args) into out_ids = sample_sequence(self.persona, self.history, self.tokenizer, self.model_checkpoint, self.args) and it works.

Could you please tell me if it's a bug here or if I ran and changed the code in a wrong way? In the table of Running ConvAI2 evaluation scripts in readme, the default parameters are the same as the one in Using the interaction script part, which look different from the defined parameters in evaluation script.

2. In the Using the training script part it was said "This model should give a Hits@1 over 79, perplexity of 20.5 and F1 of 16.5 using the convai2 evaluation script (see below)." I was wondering if this result is based on the model which is trained by the default parameters or the parameters which are given in your example (see below)?

python -m torch.distributed.launch --nproc_per_node=8 ./train.py --gradient_accumulation_steps=4 --lm_coef=2.0 --max_history=2 --n_epochs=1 --num_candidates=4 --personality_permutations=2 --train_batch_size=2 --valid_batch_size=2

I trained two models, the first one based on the default parameters and the second one based on the parameters in this example. But the first one gave better result which is correspond to "Hits@1 over 79, perplexity of 20.5 and F1 of 16.5". So I'm curious about that.

Thank you very much in advance!

[sshleifer]

Great catch, just sent #31 to fix. I don't know the answer to your training params question, unfortunately.