sonos
onnx inference is slow
I compared onnx with PPOCR on CPU and found that using python was 9 times faster than tract. I wonder if this is normal
I compared onnx with PPOCR on CPU and found that using python was 9 times faster than tract. I wonder if this is normal
onnx uses multiple threads by default. The tract is single-threaded and will of course be slower on one input. However, since tract is thread safe, I still use it in some cases.
If the task is by its nature divided into batches, for example, processing several images from a video stream, in combination with rayon there will be good speed; in my area, tract will very often be faster than onnx.
Also, if, for example, the task is to write a web server, tract can process each client independently, while the onnx model will have to be wrapped in a mutex and in this case the clients will wait for each other, which will be slow under constant load.
It all depends on the subject area.
