a bug for MNLI eval result?

[xwuShirley]

Hi there @madlag,

Thanks for your great work! It seems there is a problem for MNLI if we update text_classification/parameters.json with do_train: 0 and run following,

mkdir result
export CUDA_VISIBLE_DEVICES=0; python command_line.py finetune --json-path  text_classification/parameters.json mnli result

we get the result "eval_results_mnli.json"

{
    "eval_accuracy": 0.06622516556291391,
    "eval_loss": 15.527382850646973
}

I would really appreciate if you could double-check the code and there could be a bug? Or I must miss something.

[xiaoxiawu-microsoft]

@madlag it looks like the prediction mapping is wrong? [0,1,2], ==>-1[2,0,1]: if you go to https://github.com/huggingface/nn_pruning/blob/main/examples/text_classification/glue_xp.py#L332 and and update the prediction

            old_preds = preds.copy()
            preds = [] #[0,1,2] ==>-1[2,0,1] 
            for x in old_preds:
                if x ==2:
                    preds.append(1)
                if x ==1:
                    preds.append(0)
                if x==0:
                    preds.append(2)

we get the result "eval_results_mnli.json" { "eval_accuracy": 0.8469689251146205, "eval_loss": 15.527382850646973 }