Multi-view contrastive learning for online knowledge distillation

This project provides source code for official implementation of Multi-view contrastive learning for online knowledge distillation (MCL-OKD) and unofficial implementations of some representative Online Knowledge Distillation (OKD) methods:

• DML: Deep mutual learning (CVPR-2018)
• CL-ILR: Collaborative learning for deep neural networks (NeurIPS-2018)
• ONE: Knowledge distillation by on-the-fly native ensemble (NeurIPS-2018)
• OKDDip: Online Knowledge Distillation with Diverse Peers (AAAI-2020)
• MCL-OKD: Multi-view Contrastive Learning for Online Knowledge Distillation

We use some representative image classification networks as the role of backbone for evaluating OKD methods:

• ResNet: Deep residual learning for image recognition (CVPR-2015)
• DenseNet: Densely connected convolutional networks (CVPR-2017)
• VGG: Very deep convolutional networks for large-scale image recognition (ICLR-2015)
• HCGNet: Gated Convolutional Networks with Hybrid Connectivity for Image Classification (AAAI-2020)

Installation

Requirements

Ubuntu 18.04 LTS

Python 3.8

CUDA 11.1

PyTorch 1.6.0

Create three folders ./data, ./result, and ./checkpoint,

Perform experiments on CIFAR-100 dataset

Dataset

CIFAR-100 : download

unzip to the ./data folder

Training for baseline

python main_cifar_baseline.py --arch densenetd40k12 --gpu 0 
python main_cifar_baseline.py --arch resnet32 --gpu 0 
python main_cifar_baseline.py --arch vgg16 --gpu 0 
python main_cifar_baseline.py --arch resnet110 --gpu 0 
python main_cifar_baseline.py --arch hcgnet_A1 --gpu 0 

Training by DML

python main_cifar_dml.py --arch dml_densenetd40k12 --gpu 0
python main_cifar_dml.py --arch dml_resnet32 --gpu 0
python main_cifar_dml.py --arch dml_vgg16 --gpu 0
python main_cifar_dml.py --arch dml_resnet110 --gpu 0
python main_cifar_dml.py --arch dml_hcgnet_A1 --gpu 0

Training by CL-ILR

python main_cifar_cl_ilr.py --arch cl_ilr_densenetd40k12 --gpu 0 
python main_cifar_cl_ilr.py --arch cl_ilr_resnet32 --gpu 0 
python main_cifar_cl_ilr.py --arch cl_ilr_vgg16 --gpu 0 
python main_cifar_cl_ilr.py --arch cl_ilr_resnet110 --gpu 0 
python main_cifar_cl_ilr.py --arch cl_ilr_hcgnet_A1 --gpu 0

Training by ONE

python main_cifar_one.py --arch one_densenetd40k12 --gpu 0
python main_cifar_one.py --arch one_resnet32 --gpu 0
python main_cifar_one.py --arch one_vgg16 --gpu 0
python main_cifar_one.py --arch one_resnet110 --gpu 0
python main_cifar_one.py --arch one_hcgnet_A1 --gpu 0

Training by OKDDip

python main_cifar_okddip.py --arch okddip_densenetd40k12 --gpu 0
python main_cifar_okddip.py --arch okddip_resnet32 --gpu 0
python main_cifar_okddip.py --arch okddip_vgg16 --gpu 0
python main_cifar_okddip.py --arch okddip_resnet110 --gpu 0
python main_cifar_okddip.py --arch okddip_hcgnet_A1 --gpu 0

Training by MCL-OKD

python main_cifar_mcl_okd.py --arch mcl_okd_densenetd40k12 --nce_k 256 --gpu 0
python main_cifar_mcl_okd.py --arch mcl_okd_resnet32 --nce_k 256 --gpu 0
python main_cifar_mcl_okd.py --arch mcl_okd_vgg16 --nce_k 16384 --gpu 0
python main_cifar_mcl_okd.py --arch mcl_okd_resnet110 --nce_k 256 --gpu 0
python main_cifar_mcl_okd.py --arch mcl_okd_hcgnet_A1 --nce_k 16384 --gpu 0

• Ens : Ensemble performance with retaining all peer networks.

Perform experiments on ImageNet dataset

Dataset preparation

• Download the ImageNet dataset to YOUR_IMAGENET_PATH and move validation images to labeled subfolders

  • The script may be helpful.

• Create a datasets subfolder and a symlink to the ImageNet dataset

$ ln -s PATH_TO_YOUR_IMAGENET ./data/

Folder of ImageNet Dataset:

data/ImageNet
├── train
├── val

Training for baseline

python main_imagenet_baseline.py --arch resnet34 --gpu 0

Training by MCL-OKD

python main_imagenet_mcl_okd.py --arch mcl_okd_resnet34 --gpu 0

Ablation study on CIFAR-100 dataset

• MCL : The loss of Multi-view Contrastive Learning

• DOT : The loss of Distillation from an Online Teacher