a-PyTorch-Tutorial-to-Image-Captioning
a-PyTorch-Tutorial-to-Image-Captioning copied to clipboard
sgrvinod
BLEU-4 score @ beam size of 5 is 0.0606. result on author's model.
Hi
I have used the authors' WORDMAP and checkpoint files but after running the eval.py on these files with correct filepaths, I received a BLEU-4 score @ beam size of 5 is 0.0606. Do you think I am doing something wrong? I would really appreciate your input on this. Thank you.
Hi, Came across the same problem when trying to evaluate the model. Hope someone can offer a solution...
Hello, I just ran eval.py with my checkpoint and wordmap but received a score of 0.3028 (i.e. 30.28) with a beam size of 1 as I had reported earlier in issue #35. (The score of 29.98 on the Readme is actually on the validation set as discussed in that issue, which I hadn't updated.)
Perhaps there's a mistake somewhere?
Note: I didn't need to downgrade PyTorch to version 0.4 (the version the checkpoints were initially trained on).
I just noticed you mentioned that it happened at a beam size of 5. At this beam size, I get a score of 0.3329 (i.e. 33.29).
@ada-dl Got the same number 0.0606. The reason is that, I used from skimage.transform import resize as imresize and img = imresize(img, (256, 256). The default behavior of resize is to convert original range 0-255 to 0-1, so you should use img = imresize(img, (256, 256),preserve_range=True) instead. Everything was normal after the change
@sddduuu Hello, I want to know if the line img = imresize(img, (256, 256) in utils.py needs to be modified, which is a preprocessing step for the dataset?
@sddduuu Hello, I want to know if the line img = imresize(img, (256, 256) in utils.py needs to be modified, which is a preprocessing step for the dataset?
Hello, I also occurred the BLEU-4 pretty low issue (around 0.14) and read this post, change to add preserve_range=True as below in utils.py and caption.py, then the BLEU-4 get to over 20 and caption.py make sense.
from skimage.transform import resize
img = resize(img, (256, 256), anti_aliasing=True, preserve_range=True, mode='constant')
@Nanochuan Hello, Can you completely reproduce the author's scores? The imresize function has been deprecated since scipy 1.3.0, but I noticed some issues when directly using the resize function. What I want to know now is: is your approach feasible?
@Nanochuan Hello, Can you completely reproduce the author's scores? The imresize function has been deprecated since scipy 1.3.0, but I noticed some issues when directly using the resize function. What I want to know now is: is your approach feasible?
Hi! Yes, with the change I mentioned above, I am able to get similar result like author. So far I run with ResNet-101 as encoder and run 4 Epoch, the BLEU-4 score is 22.33. And comments generated from picture by caption.py is quite reasonable.