Does it support quantization?

[yrik]

I want to improve FPS on Raspberry PI. Does it make sense to look into 16 or 8 bit quantization? Would be possible to run it with this code?

[Qengineering]

It all depends on the quantization tool of the ncnn framework.

As you no doubt know, deep learning models consist of many layers with different operations. As time goes by, many new operations are introduced by the deep learning community, giving programmers the noble task of implementing them in their frameworks. Here comes the question of whether an operation can be transferred to INT8 (or INT16). Sometimes it just can't. Sometimes only by a few tricks. YoloV8 is a relatively new model, so don't be surprised if quantization fails. Look for an example of your adventure on this site: https://blog.csdn.net/weixin_45829462/article/details/119787840

[yrik]

Thank you for the detailed response. The article by the link is very interesting. I was thinking to use quantization from Ultralytics (they have it out of the box) and then exporting it further. But it seems it's much more adventurous task to solve.

[Qengineering]

Keep in mind that exporting Ultralytics YoloV5 or YoloV8 can be an adventure when it comes to ncnn. See: https://github.com/Tencent/ncnn/discussions/4541 Especially the already quantized version are notorious issue generators.

[yrik]

Thanks! How would you recommend overall to approach the topic of further speed up of this model on raspberry pi?

[Qengineering]

1. Try quantization, just as you suggested. Keep in mind that it can fail.
2. Try overclocking (simple and works always).
3. If possible reduce input size 320x320 is faster than 416x416, which on its turn is faster than 640x640 (must be dividable by 32)
4. If possible try another model like YoloX.