Dataset with human data

[vishaal27]

Hey,

Thanks for the great work: I enjoyed reading the paper, and your proposed iterative improving strategy seems super interesting.

I am interested in the human data you collected (DecomposedCaptions4k with 24960 human annotations), is there a planned release date for this? Also a couple more related questions:

1. I could not find the supplementary material of your paper online (it wasn't attached to the arxiv paper), are you planning an update to arxiv; I would be super interested in going over the supp. mat.
2. For the human data, did you run two separate experiments? One for the preference study with two images and one prompt displayed, and another for single image-single prompt alignment?

Thanks!

[1jsingh]

Hi,

Thanks for your interest and positive comments about our paper. We plan to release the code and Decomposed-Captions-4k data together after refactoring to allow for easy use in downstream benchmarks. We currently plan to release the same by end August / early September.

Also regarding the other questions:

1. I could not find the supplementary material of your paper online (it wasn't attached to the arxiv paper), are you planning an update to arxiv; I would be super interested in going over the supp. mat.

Thanks for pointing that! We will update the arxiv version to also include the supp. material.

2. For the human data, did you run two separate experiments? One for the preference study with two images and one prompt displayed, and another for single image-single prompt alignment?

Yes, pairwise comparison scores are reported using a separate user-study, wherein subjects are shown a pair of images and asked to select the one with better alignment.

[vishaal27]

Great, looking forward to the code release!

[FlyFOV]

Nice solution to alignment evaluation. Just wander wether the code/dataset will be released this Month~

[vishaal27]

Hi @1jsingh, just checking to see if you had any updates about releasing the human annotations you collected?