Improving Knowledge Distillation via Regularizing Feature Norm and Direction

Pytorch implementation for the paper: Improving Knowledge Distillation via Regularizing Feature Norm and Direction.

0. Framework

The ND loss, that regularizes the Norm and Direction of the student features, be applyed the embedding features, which is defined as the output at the penultimate layer before logits.

1. Main Results

1.1 CIFAR-100

** represents our reproduced based on the official code DKD.

1.2 ImageNet-1k

• Comparisons with State-of-the-art Results

• Benefit from larger teacher models

With the teacher capacity increasing, KD++, DKD++ and ReviewKD++ (red) is able to learn better distillation results, even though the original distillation frameworks (blue) suffers from degradation problems. The student is ResNet-18, and reported the Top-1 accuracy (%) on the ImageNet validation set. All results are the average over 5 trials.

• Distillation between CNN and Transformer

  • ViT $\rightarrow$ ResNet

    student	teacher	student	teacher	KD	DKD	KD++	DKD++
    Res-18	ViT-S	69.76	74.64	71.32	71.21	71.46	71.33
    Res-18	ViT-B	69.76	78.00	71.63	71.62	71.84	71.69

  • CNN $\rightarrow$ ViT

    We follow the design of DeiT: add a distillation token to the input sequences, and use the mean of cls token and dis token to evaluating. We add ND loss to this framework to improve performance of distillation.

    student	teacher	student	teacher	DeiT	DeiT++	checkpoint
    DeiT-B	RegNetY-16GF	81.8	82.9	83.4	83.6	baiduyun (7hnj)

    Notes

    • The distillation hyper-parameters are slightly different from the original DeiT. We provide recipe as shown in the table.
      

    The recipe follow MAE, expect batch size, because lack gpu, and these parameters are not tuned. When using this recipe without ND, a distillation result of 83.35 is obtained, similar to DeiT.

    • The distillation process is very slow, it takes about 6.5 days to train on 4*A100, and half as slow as training ViT-b without distillation.

2. Training and Evaluation

2.1 CIFAR-100 Classification

Please refer to CIFAR for more details.

2.2 ImageNet Classification

Please refer to ImageNet for more details.

2.3 COCO Detection

Please refer to Detection for more details.

3. Citation

If you use ND in your research, please consider citing:

@misc{wang2023improving,
      title={Improving Knowledge Distillation via Regularizing Feature Norm and Direction}, 
      author={Yuzhu Wang and Lechao Cheng and Manni Duan and Yongheng Wang and Zunlei Feng and Shu Kong},
      year={2023},
      eprint={2305.17007},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}