Experiment

[alibabadoufu]

Hi, thanks for the cool work!

I am wondering if you could construct the negative captions with another alternative:

1. collecting all the noun words in the training set
2. randomly substituting the noun word for a pair of image-caption sample with the collection. (remove those synonyms by re-ranking for training)

I am trying to guess how important is the context in this case.

[BigRedT]

Hi @alibabadoufu,

That's a great experiment suggestion and is certainly worth trying! Just one thing to note is that this "random-substitution" baseline while not involving context in substitute sampling, will still involve context during featurization (since we use a language model to compute contextualized word representations). But I agree, this is still a good test to see how important is context during sampling.

You can actually try this easily by modifying sample_neg_bert.py file in case I don't get a chance to run this. I would be very curious to know the results if you do try this!

Thanks, Tanmay