Fine tuning Virtex for image captioning

[freeIsa]

Hi there, I am aware that Virtex used image captioning as a pretraining task and not as the "final goal", but I was wondering whether one could go on fine-tuning the pretrained model (e.g. bicaptioning_R_50_L1_H2048) with additional COCOcaptions-like data in order to get an improved captioning model. Has anyone tried that or does anyone have any suggestion how to do it? Can any of the scripts in the repository be used/adapted for fine-tuning existing models? Thanks a lot! :)

[bermeitinger-b]

You could use the pretrained ResNet-50 and then, similar to the pretraining layout, add the Transformers back in. But now, you freeze ResNet-50 and use more decoder layers instead of one.