Add cskd loaders for aircrafts and dog datasets

[hlml]

Hi Ahmed,

I've added a cs-kd supported version of the aircrafts and the dog120 dataset. Can you give it a try to see if it works for you? You will need to change some of the hard coded values as well as enable cs-kd.

When I run this on my side, I'm seeing a consistent 2-4% decrease in performance for the dog dataset for both smoothing and cs-kd, and a 2-4% increase for smoothing in the aircrafts dataset. This may be due to just variance. But for cs-kd on aircrafts, I'm seeing a 6% increase in performance. I wanted to check if there is something different with the cs-kd loader you used to cause this performance discrepancy, and if there's some data processing you may have done specifically for the dog dataset (everything is resized to 256x256 currently).

Thanks! Hattie

[ahmdtaha]

Hi Hattie,

I checked-out your commit, but I can't run it because it leverage datasets.ImageFolder. My datasets doesn't have train, val, test directories. Accordingly, I can't run/evaluate your code. It is convenient to use ImageFolder but this is not a good choice on the long-run. For instance, if you want to train or eval on a subset of your dataset, you have to create a new directory and copy subset of images to this new directory. In contrast, if you use csv files, you only create a new csv file with a subset of image names -- no need to copy images.

Regarding the performance degradation, I am not sure what is wrong. I see you are using CS_KD authors implementation[1]. The images inside my dogs directory are Not normalized to 256x256. Sorry for such inconsistency. As mentioned previously, I don't remember resizing all datasets.

For stanford_dogs, If the original images didn't fix the cs_kd issue, I want you to try something crazy! Assuming, batch-size=4, the current PairBatchSampler returns a batch with labels as follows: [c_1, c_2,c_3,c_4, c_1, c_2,c_3,c_4]. Can you please change that to [c_1,c_1,c_2,c_2,c_3,c_3,c_4,c_4]. Basically, change yield batch_indices + pair_indices to yield list(itertools.chain(*zip(batch_indices,pair_indices ))). Keep the trainers.default_cls as it is. I know that doesn't make a lot of sense, but I hope you try it and let me know if it makes any difference in performance.

Thanks [1] https://github.com/alinlab/cs-kd

[hlml]

Hi Ahmed! I tried the change u suggested for PairBatchSampler, it seems to decrease performance by ~2% compared to the original PairBatchSampler, but still 2-3% higher than the reported result in the paper. It does seem to bring the difference down to a level that's more likely to just be noise / environment differences though.

[ahmdtaha]

Hi Hattie,

I am getting mixed signals here -- "2-3% higher than the reported result in the paper". Which version achieves this 2-3 higher performance?

[hlml]

Sorry for the confusion. In the original post, I mentioned that the cs-kd implementation got ~6% higher results than reported in the paper. After trying your suggested change, the difference in performance is ~3% higher than the paper. Hope that makes more sense.

[ahmdtaha]

That makes more sense. It is good that your cs-kd implementation achieves ~6% higher results. I am no longer confident about my cs-kd baseline. I used multiple samplers in my paper. I used cs-kd sampler for classification experiments (Sec 4.1) but then another sampler for my retrieval experiments (Sec 4.2). I might have confused these samplers while switching back and forth between classification and retrieval experiments.

The good news is that KE works on top of various baselines. If your cs-kd gives better results than mine, that might push KE+CS-KD further. Does that make sense?