Performance on KonIQ10K and CLIVE

[zwx8981]

Hi, can you describe in details the evaluation protocols of CLIVE and KonIQ?

As reported in the paper, your model achieves 0.784 and 0.788 on CLIVE and KonIQ, respectively.

I use your pre-trained model to test on these two databases. In my experiments, images are randomly divided into training and testing sets under a 80%/20% ratio. But I can only get median SRCC across ten sessions on the test sets of CLIVE and KonIQ of 0.719 and 0.722, which are far from the numbers reported in your paper.

[baidut]

Hi, the reported results are produced using one random 80%-20% split. Yes, it would be better to use mean/median performance across different splits. The reason for the performance gap might be that the released model was trained on images we can release, while the model we reported the performance was trained on a larger image set. We crawled some Facebook profile images and labeled them at the beginning but when we are releasing the database, we were told that although the profile images are from public domains, it might bring some legal issues if we publish them. So we retrained the model using the released dataset.

I thought about releasing the original model but it was trained on a different split and more images. To make sure that researchers can make a fair comparison. we provide a fixed random split for the published dataset.

What I can do at this point is, to test the original model on multiple splits of CLIVE and KonIQ as well and compared them with the numbers you reported. Please allow me some time to do that.

Thank you for your time!