CLIP_prefix_caption
CLIP_prefix_caption copied to clipboard
Evaluate problem
It's a very nice work,can you provide the code for evaluation it? I don't know how to evaluate it. Thank you!
Hi @wytszhuzhu, we have used the Oscar evaluation code to evaluate over COCO
I Hope this is helpful
oh! thank you very much!!!
------------------ 原始邮件 ------------------ 发件人: "rmokady/CLIP_prefix_caption" @.>; 发送时间: 2022年3月25日(星期五) 上午6:21 @.>; 抄送: "pl0t-k ???? ?? @.@.>; 主题: Re: [rmokady/CLIP_prefix_caption] Evaluate problem (Issue #33)
Hi @wytszhuzhu, we have used the Oscar evaluation code to evaluate over COCO
I Hope this is helpful
— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>
Hello! I have some questions about settings. I notice that the default training script does not use normalize_prefix:
python train.py --only_prefix --data ./data/coco/oscar_split_ViT-B_32_train.pkl --out_dir ./coco_train/ --mapping_type transformer --num_layres 8 --prefix_length 40 --prefix_length_clip 40
While the infernce code, transformer_inference.ipynb
, uses normalize_prefix:
prefix = prefix / prefix.norm(2, -1).item()
So, for COCO caption dataset, which is better, normalize prefix or not? Thanks!
@wytszhuzhu -- I've added basic eval support in my branch: https://github.com/thandal/CLIP_prefix_caption ''' python3 train.py --only_prefix --data data/coco/oscar_split_ViT-B_32_val.pkl --out_dir ./coco_evaluate_local_2/ --mapping_type transformer --num_layers 8 --prefix_length 40 --prefix_length_clip 40 --model_path=models/transformer/coco_prefix-009.pt --evaluate ''' Its definitely a little clunky, but I also have a few scripts to massage the results into the type of files used by coco-eval.
Unfortunately, I haven't been able to reproduce the results in the paper yet!
val_karpathy: {'bleu': [0.6574263790124814, 0.46054883354763715, 0.3079421708904345, 0.20412719661970102], 'cider': 0.5766721258280709, 'meteor': 0.23068635129255388, 'rouge': 0.46257322083431907, 'spice': 0.15591030077420115, 'wmd': 0.5299810088210446} test_karpathy: {'bleu': [0.6548157984748666, 0.4597153137744317, 0.3103062426657983, 0.2078126593622194], 'cider': 0.5843739503309977, 'meteor': 0.23188037724901417, 'rouge': 0.4614305714563363, 'spice': 0.15830000031265373, 'wmd': 0.5291680687998455}
hello,@wytszhuzhu,Have you solved the problem?sos
@thandal hello,i run your train.py,then In my checkpoints floder a file named evaluate.json is generated, and then what do I do to get scores for various metrics?Thanks
I think I used (from my branch) captions_to_gt_for_eval.py -- to convert clipcap-style caption files to ground-truth coco-caption eval style.
And then ran the eval code from https://github.com/EricWWWW/image-caption-metrics, with a patch: https://github.com/EricWWWW/image-caption-metrics/compare/master...thandal:image-caption-metrics:patch-1
@thandal Thank you ,buddy, I'll try that in a minute,I'm sorry I forgot to give a Star
I think the possible reason could be that they train on both the train and val set while testing only on test set.
can anyone could tell me from where I can get oscar_split_ViT-B_32_train.pkl file
can anyone could tell me from where I can get oscar_split_ViT-B_32_train.pkl file
See https://github.com/rmokady/CLIP_prefix_caption/blob/main/README.md#coco-training
can anyone could tell me from where I can get oscar_split_ViT-B_32_train.pkl file
See https://github.com/rmokady/CLIP_prefix_caption/blob/main/README.md#coco-training
Thanks got it