AB_distillation
AB_distillation copied to clipboard
Knowledge Transfer via Distillation of Activation Boundaries Formed by Hidden Neurons (AAAI 2019)
Knowledge Transfer via Distillation of Activation Boundaries Formed by Hidden Neurons
Official Pytorch implementation of paper:
Knowledge Transfer via Distillation of Activation Boundaries Formed by Hidden Neurons (AAAI 2019).
Slides and poster are available on homepage
Environment
Python 3.6, Pytorch 0.4.1, Torchvision
Knowledge distillation (CIFAR-10)
cifar10_AB_distillation.py
Distillation from WRN 22-4 (teacher) to WRN 16-2 (student) on CIFAR-10 dataset.
Pre-trained teacher network (WRN 22-4) is included. Just run the code.
Transfer learning (MIT_scenes)
MITscenes_AB_distillation.py
Transfer learning from ImageNet pre-trained model (teacher) to randomly initialized model (student).
Teacher : ImageNet pre-trained ResNet 50
Student : MobileNet or MobileNetV2 (randomly initialized model)
Please change base learning rate to 0.1 for MobileNetV2.
MIT_scenes dataset should be arranged for Torchvision ImageFolder function.
Train set :
$dataset_path / train / $class_name / $image_name
Test set :
$dataset_path / test / $class_name / $image name
and run with dataset path.
MobileNet
python MITscenes_AB_distillation.py --data_root $dataset_path
MobileNet V2
python MITscenes_AB_distillation.py --data_root $dataset_path --network mobilenetV2
Other implementations
Tensorflow: https://github.com/sseung0703/Knowledge_distillation_methods_wtih_Tensorflow
Citation
@inproceedings{ABdistill,
title = {Knowledge Transfer via Distillation of Activation Boundaries Formed by Hidden Neurons},
author = {Byeongho Heo, Minsik Lee, Sangdoo Yun, Jin Young Choi},
booktitle = {AAAI Conference on Artificial Intelligence (AAAI)},
year = {2019}
}