show-attend-and-tell
show-attend-and-tell copied to clipboard
alecwangcq
no validation loss
I'm pretty confused because there's no loss calculated in validation phase. Normally there should be some validation loss/accuracy/perplexity.
Do we really need to calculate validation loss for the generated captions?
Yes, you are right. But I did not implement the validation part, which is test() in main.py. In general, it will be tested on validation set for computing scores such as: bleu, meteor, etc. Please refer to https://github.com/tylin/coco-caption.
I looked up other implementations and they didn't include the validation loss, either. But indeed those evaluation metrics are pretty critical for us to tell the performance of the model.
I implemented the test() and include the validation loss. But unfortunately the validation loss could be very high even if the BLEU-n scores and other metrics look completely normal.
So I guess it could be cool if you can check the validation loss ;)
How did you compute the validation loss? Feed the caption in a teacher-forcing manner, and then compute the loss, just like what we did for training?
Tried two ways. One way is generating texts with teacher- forcing while the other is without teacher-forcing but is constrained to have the same length as the target captions.
However I think both ways are problematic because the generating of texts is not spontaneous.