yolo2_light
yolo2_light copied to clipboard
INT8 cudnn with 1st layer return error
Currently Yolo2_light supports quantization for some specified layers, Not the 1st layer, Not the layer before yolo(region), and Not the size=1 layer.
And I tried to make a full INT8 quantization, but however, when I changed the 1st layer on Yolov3-tiny, it says
CUDNN failure
Error: 3 - CUDNN_STATUS_BAD_PARAM
my questions are
- is that possible to quantize with all the layers? don't care about mAP lose at this stage.
- is that possible to fixed the CUDNN error?
The last layer must be fp32, since it is used for object coordinates. May be it can be int8 only if we use much more anchors (initial w,h defined with lower step) and higher network resolution (initial x,y defined with lower step).
- It seems you changed something incorrect.