Need help for replicating Low Resource Track.

[sai-prasanna]

I followed the instructions for the low-resource track. After the DAE step, I get 0.33 score with evaluate.py on the best model. But on training with the 3k dev set and evaluating, the score becomes very low. I tried increasing the number of epochs, lowering learning rates. But I am unable to even retain the F score from the DAE pre-training. Need help to replicate the train step.

[hammouse]

We used 5e-4 learning rate for the DAE pre-training. We also did not reset the optimizer when we start the training step.

[sai-prasanna]

@hammouse As a clarification, what is the single model F-score, final validation loss you get for low resource track after training on 3k dev set? Can you share the training log?

I didn't change any learning rate or other hyper parameters, just used the track 3 training after pretraining commands you have provided.

My loss starts at 15.0 and perplexity is very high and as the training dataset is small (just 3k) it ends fairly quickly. And all the trained epoch checkpoints on evaluation, result is very poor since model generates incoherent text after training on 3k.

[sai-prasanna]

@hammouse Can you release a trained model for the third track, or just the logs for training ?

[hammouse]

At the end of the pre-training, the validation loss reached 2.976. After the training, the validation loss reached 2.705.

[niucheney]

Because there is a bug in train.py. I fixed the bug and had replicated the experiment.
In "def train " function, I fixed the bug as follows. `def train(databin_path, model_config, ckpt_dir, restore_ckpt, ngpu): xxx

if restore_ckpt is not None:
    finetune_ckpt =util.change_ckpt_dir(restore_ckpt, ckpt_dir)
    logging.info(f"[Train] copy the ckpt {restore_ckpt} into {finetune_ckpt}")
    os.system(f"cp {restore_ckpt} {finetune_ckpt}")
    finetune_ckpt = os.path.basename(util.change_ckpt_dir(restore_ckpt, ckpt_dir))
    prompt += f"--restore-file {finetune_ckpt} "
os.system(prompt)`