Table 6 question

[rkdckddnjs9]

thank you for your excellent works!

In table 6, the student's parameters and FLOPs do not change. Is this because the student features are put into the head without going through the denoising process?

[hunto]

Hi @rkdckddnjs9 ,

Yep. The denoising process is not involved in the inference of student after KD training. During KD, the student head still uses the original feature (not the denoising one) for prediction. You can consider the denoising as a stronger aligment module to trasnform the student features, and the transformed features are ONLY used for distillation.

[omglet1]

Oh @hunto , This is amazing. You mean the diffusion and noisy adapter only be used as a alignment module to train the backbone of student in the stage of KD, right? During inferecing, you do not use diffusion model and the input image only be processes by the backbone and head of student. I'm curious about your opinion on this issue. if you use the diffusion and noisy adapter to transform the student feature in the stage of inference, is it possible or what would be affectd?

[hunto]

Oh @hunto , This is amazing. You mean the diffusion and noisy adapter only be used as a alignment module to train the backbone of student in the stage of KD, right? During inferecing, you do not use diffusion model and the input image only be processes by the backbone and head of student. I'm curious about your opinion on this issue. if you use the diffusion and noisy adapter to transform the student feature in the stage of inference, is it possible or what would be affectd?

Yes. Involving the diffusion model to also transform the feature in inference is an interesting idea, but I didn't try it before in the paper since it may cause some issue of unfair comparisons in student architectures.