About punctuations

[zml110120]

Dear author, It is a fantastic work! But there are some problems when i use it. I need to generate captions with punctuations, just like 【There are a, b and c. I like them.】. I want to output captions containing periods and commas. How can i do this?

[abaybektursun]

I asked the same question here: https://github.com/OFA-Sys/OFA/issues/205 I solved mine by editing the code and removing punctuations from transtab. You have to re-train the model. The way they designed their captioning is to do short one sentence descriptions.

[logicwong]

Yes, you can do as Abay says. We remove these punctuations because most of the descriptions in COCO are single sentences, and punctuation is not helpful for CIDEr calculation.

[zml110120]

@abaybektursun @logicwong Thanks for your help, but there are still some problems. I used the breakpoint where you said it, but it doesn't seem to work. transtab = str.maketrans({key: None for key in string.punctuation})

I run the code in Pycharm IDE. Here is my running script

/usr/zmlCV/OFA/dataset/caption_data4/cp_caption_train.tsv,/usr/zmlCV/OFA/dataset/caption_data4/cp_caption_val.tsv --selected-cols=0,1,2 --bpe-dir=utils/BPE --user-dir=ofa_module --restore-file=checkpoints/ofa_base.pt --reset-optimizer --reset-dataloader --reset-meters --save-dir=/usr/zmlCV/OFA/dataset/caption_data4/stage1_checkpoints --task=caption --arch=ofa_base --criterion=adjust_label_smoothed_cross_entropy --label-smoothing=0.1 --batch-size=8 --update-freq=4 --encoder-normalize-before --decoder-normalize-before --share-decoder-input-output-embed --share-all-embeddings --layernorm-embedding --patch-layernorm-embedding --code-layernorm-embedding --resnet-drop-path-rate=0.0 --encoder-drop-path-rate=0.1 --decoder-drop-path-rate=0.1 --dropout=0.1 --attention-dropout=0.0 --weight-decay=0.01 --optimizer=adam --adam-betas="(0.9,0.999)" --adam-eps=1e-08 --clip-norm=1.0 --lr-scheduler=polynomial_decay --lr=1e-5 --max-epoch=55 --warmup-ratio=0.06 --log-format=tqdm --log-interval=10 --fixed-validation-seed=7 --no-epoch-checkpoints --keep-best-checkpoints=1 --save-interval=1 --validate-interval=1 --save-interval-updates=500 --validate-interval-updates=500 --maximize-best-checkpoint-metric --max-src-length=99 --max-tgt-length=99 --find-unused-parameters --freeze-encoder-embedding --freeze-decoder-embedding --add-type-embedding --scale-attn --scale-fc --scale-heads --disable-entangle --num-bins=1000 --patch-image-size=384 --drop-worst-ratio=0.2 --drop-worst-after=2500 --fp16 --fp16-scale-window=512 --num-workers=0

[logicwong]

@zml110120 set transtab = str.maketrans({key: None for key in string.punctuation if key != ','}) or just remove this line.

[abaybektursun]

@zml110120 Also make sure to do the same in /data/mm_data/caption_dataset.py, tasks/mm_tasks/caption.py, and utils/eval_utils.py