edgeai-mmdetection
edgeai-mmdetection copied to clipboard
TexasInstruments
the rusult of QAT model in .onnx and .pt are much different
When training done with fp32, I convert the .pt model to .onnx, and I respectively use them to predict one pic, and the results are very close. While when I QAT this model, and doing the same step, the results are much different. Is this something wrong during the convert process? Should there be any more attentions when convert a QAT model to onnx? Thx
When training done with fp32, I convert the .pt model to .onnx, and I respectively use them to predict one pic, and the results are very close
QAT onnx model from QuantTrainModule() is meant to be used with TIDL (i.e. with onnxruntime-tidl and with parameters set correctly for QAT model calibration) If you use regular onnxruntime, it would try to do a float inference in this QAT model and the result would not be good.
When training done with fp32, I convert the .pt model to .onnx, and I respectively use them to predict one pic, and the results are very close
QAT onnx model from QuantTrainModule() is meant to be used with TIDL (i.e. with onnxruntime-tidl and with parameters set correctly for QAT model calibration) If you use regular onnxruntime, it would try to do a float inference in this QAT model and the result would not be good.
Thx, I get. One more question: eg. I'm using yolox of edgeai-mmdetection to do QAT, maybe like 12 epoch, after training done it will do evaluation on test set, if I get a map50 like 0.80, then I convert this model to a .onnx, and I used it on onnxruntime-tidl, will the result be same or close to 0.80?
That is the expectation, getting exact accuracy involves a lot of aspects including using the correct confidence threshold, top-k boxes, input pre-processing etc. So, do don't think it will be easy to match the accuracy exactly.
That is the expectation, getting exact accuracy involves a lot of aspects including using the correct confidence threshold, top-k boxes, input pre-processing etc. So, do don't think it will be easy to match the accuracy exactly.
Export QAT model just using regular method like torch.onnx.export is ok?
Yes - you can see details here: https://github.com/TexasInstruments/edgeai-torchvision/blob/master/docs/pixel2pixel/Quantization.md#how-to-use--quanttrainmodule
Yes - you can see details here: https://github.com/TexasInstruments/edgeai-torchvision/blob/master/docs/pixel2pixel/Quantization.md#how-to-use--quanttrainmodule
I'm very new to this area, now I have a trained QAT model, and the map I get during QAT on test set is like 0.68 (only drop 0.02 compared to fp32 pretrained model), I want to double-check the infer result, is there any doc about how to use this QAT model correctly like using onnxruntime-tidl u mentioned?
Accuracy benchmark is provided via: https://github.com/TexasInstruments/edgeai-benchmark
See the instructions to compile custom models: https://github.com/TexasInstruments/edgeai-benchmark/blob/master/docs/custom_models.md