keras-FP16-test
keras-FP16-test copied to clipboard
Evaluating deep learning models with float16 dtype in Keras, float16 inference
Keras float16 vs float32
Requirements
keras==2.1.0
tensorflow==1.8.0
opencv==3.2.0
pycocotools, BeautifulSoup4, lxml, tqdm
How to set float16 compute mode.
from keras import backend as K
K.set_floatx('float16)
Image Classification Part
Usage
Firstly, download ImageNet val data and model pre-trained weights file.
An example for testing vgg16 with float16.
python eval_image_classification.py --model='vgg' --dtype='float16'
An example for testing mobilenet with a width multiplier 1.0.
python eval_image_classification.py --model='mobilenet' --dtype='float16' --alpha=1.0
ImageNet Datatset
ImageNet val data provided by aaron-xichen, sincerely thanks to aaron-xichen for sharing this processed ImageNet val data.
Results
TOP1 acc and TOP5 acc on ImageNet val data.
Pre-trained weight files are downloaded from deep learning models, DenseNet121 weight file is downloaded from DenseNet-Keras Squeezenet weight file is downloaded from keras-squeezenet
There is a backup of weights in baiduyun(百度云)
float32 | float16 | diff | ||||
Model | Top1 acc | Top5 acc | Top1 acc | Top5 acc | Top1 acc | Top5 acc |
VGG16 | 0.70786 | 0.89794 | 0.7082 | 0.89802 | 0.00034 | 0.00008 |
ResNet50 | 0.74366 | 0.91806 | 0.70508 | 0.89466 | -0.03858 | -0.0234 |
Inceptionv3 | 0.76518 | 0.92854 | 0.765 | 0.92854 | -0.00018 | 0.00 |
Inception-ResNet | 0.789 | 0.94426 | 0.7888 | 0.94436 | -0.0002 | 0.0001 |
DenseNet121 | 0.74234 | 0.91868 | 0.74206 | 0.91868 | -0.00028 | 0.000 |
Xception | 0.77446 | 0.93618 | 0.77392 | 0.93596 | -0.00054 | -0.00049 |
Squeezenet | 0.52294 | 0.76312 | 0.52172 | 0.76226 | -0.00122 | -0.00086 |
MobileNet-1-0 | 0.69856 | 0.89174 | 0.6966 | 0.8898 | -0.00196 | -0.00194 |
MobileNet-7-5 | 0.67726 | 0.87838 | 0.6726 | 0.87652 | -0.00466 | -0.00186 |
MobileNet-5-0 | 0.6352 | 0.85006 | 0.62944 | 0.84644 | -0.00576 | -0.00362 |
MobileNet-2-5 | 0.5134 | 0.75546 | 0.50272 | 0.74648 | -0.01068 | -0.00898 |
Object Detection Part
Usage
Firstly, download VOC2007 test set and COCO2017 val set, COCO2017 val set annotations datasets, then extract them and modify the path in script.
Secondly, download SSD pre-trained weights and put them in 'weights' directory.
SSD300 VOC weights, SSD300 COCO weights, SSD512 VOC weights, SSD512 COCO weights
The method for converting the original YOLOv3 model to a keras model can be found in this repo.
An example for evaluating SSD300 on VOC2007 test set
python eval_object_detection.py --model='ssd300' --dtype='float16' --eval-dataset='voc2007'
Results
Notice that SSD models suffer significant accuracy loss.
SSD results on VOC2007 test set
mAP | |||
Model | float32 | float16 | diff |
SSD300 | 0.782 | 0.769 | -0.013 |
SSD512 | 0.91 | 0.868 | -0.042 |
The AP of each category can be found in this doc
SSD and YOLOv3 results on COCO val2017.
mAP | |||
Model | float32 | float16 | diff |
SSD300 | 0.424 | 0.374 | -0.050 |
SSD512 | 0.481 | 0.448 | -0.033 |
YOLO320 | to do | to do | to do |
YOLO416 | to do | to do | to do |
YOLO608 | to do | to do | to do |
Semantic Segmentation Part
In this part, I evaluate semantic segmentation with float16 dtype.
U-net is adopted in this test.
HumanParsing-Dataset is adopted in this test.
The tested models are trained by my-self. Training details can be found in this repo: Person-Segmentation-Keras.
Usage
For person segmentation (binary classification) task.
python eval_segmentation.py --model='unet' --dtype='float16' --nClasses=2
For human parsing (multi-class classification) task.
python eval_segmentation.py --model='unet' --dtype='float16' --nClasses=5
Results
Person segmentation
mIoU | |||
Model | float32 | float16 | diff |
Unet | 0.8920 | 0.8918 | -0.0002 |
Human parsing
mIoU | ||||
Part | float32 | float16 | diff | |
Unet | head | 0.66476 | 0.66463 | -0.00013 |
upper body | 0.48639 | 0.48640 | 0.00001 | |
both hands | 0.27016 | 0.27005 | -0.00011 | |
lower body | 0.66536 | 0.66520 | -0.00016 | |
mean | 0.52167 | 0.52157 | -0.0001 |
PointNet Evaluation
PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation
In this evaluation, I use a pre-trained PointNet reference to PointNet-Keras.
Results
3D classification
acc | |||
Model | float32 | float16 | diff |
PointNet_cls | 0.87824 | 0.87784 | -0.0004 |