plkmo
Loss Curve
Hi @plkmo !
Great work!
Do you have the loss curve available from your trainings (pre-training and semeval training) so we can check if our experiments are matching yours?
Thanks!
Hi, yes the training loss curve for semeval training is in the results folder. Please note that the MTB model has been updated and hence the old loss curve for MTB pretraining should be ignored
Yup, will do so once I have the available GPU compute to satisfactorily pre-train it on suitable data.
I'm reopening since we're still discussing this :sweat_smile:
I got these losses. Do you think they are ok?
Looks good, I also got something like this with cnn dataset. But note that the loss consists of lm_loss + MTB_loss. From what I can see, lm_loss seems to decrease much more than MTB loss.
If you can, try a larger dataset for MTB pre-training, as cnn dataset might be too small. Eg. the paper used wiki dumps data which is huge.
@plkmo From what I could see, you weren't able to get good results from MTB using cnn either, right? I did a pretraining and applied it on the task afterwards, and the results were quite worse than using bert alone.
Yeah, no good results pretraining MTB based on CNN dataset so far. Best is to directly fine-tune using pre-trained BERT.
sorry to bother you, but when I run the program first day, it did work. But the next day there was a problem with the program when I run the program.
it said IndexError: ('list index out of range', 'occurred at index 47') please help , I really appreciate if you can have a look on it , sorry for you time thanks a lot
prog-bar: 100%|██████████| 8000/8000 [00:01<00:00, 4026.81it/s]
prog-bar: 1%| | 96/8000 [00:00<00:00, 13751.82it/s]
Traceback (most recent call last):
File "C:/article/MTB/main_task.py", line 49, in
Yeah, no good results pretraining MTB based on CNN dataset so far. Best is to directly fine-tune using pre-trained BERT.
My result of MTB pretraining based on CNN dataset is bad too, and pretraining task takes a long time. I wonder how much time do you take to pretraining MTB with a good result?
@zjucheri : I found that MTB training on the CNN data beyond about 9 epochs degraded performance on FewRel. The key to better performance on is probably to use a larger (and perhaps more relevant or at least generic) data set such as WikiPedia.
@plkmo: Thanks for sharing your rather nice code :)
