OFA icon indicating copy to clipboard operation
OFA copied to clipboard

Published 20 hours ago • verified publisher icon OFA-Sys

Question about data processing in pretrain data.

Open JJJYmmm opened this issue 11 months ago • 1 comments

Why the following actions were taken. Is there anything special about cc12m I missed?

https://github.com/OFA-Sys/OFA/blob/a36b91ce86ff105ac8d9e513aa88f42b85e33479/data/pretrain_data/unify_dataset.py#L321-L323

Looking forward to your reply.

JJJYmmm avatar Mar 10 '24 09:03 JJJYmmm

Another question. When computing the loss AdjustLabelSmoothedCrossEntropyCriterion, sample_patch_num is added into the model input(sample[0], which I think is correspond to sample_v1, the vision-language data)

https://github.com/OFA-Sys/OFA/blob/a36b91ce86ff105ac8d9e513aa88f42b85e33479/criterions/label_smoothed_cross_entropy.py#L177-L178

It seems that sample_patch_num can select fixed number of image features. So why it's just used in VL data?

https://github.com/OFA-Sys/OFA/blob/a36b91ce86ff105ac8d9e513aa88f42b85e33479/models/ofa/unify_transformer.py#L759-L769

JJJYmmm avatar Mar 10 '24 13:03 JJJYmmm