rf-detr
rf-detr copied to clipboard
roboflow
predict with onnx
Description
This PR adds support for ONNX-based inference in RF-DETR, enabling faster model execution compared to PyTorch CPU inference. This change introduces an alternative inference path using the onnxruntime backend.
Ticket : https://github.com/roboflow/rf-detr/issues/64
Type of change
- [x] New feature (non-breaking change which adds functionality)
- [ ] This change requires a documentation update
How has this change been tested, please provide a testcase or example of how you tested the change?
Python test file
import time
import cv2
import numpy as np
from PIL import Image
import supervision as sv
from rfdetr.detr import RFDETRLarge
from rfdetr.util.coco_classes import COCO_CLASSES
def predict(model, image_path, im_save):
image = Image.open(image_path)
start = time.time()
detections = model.predict(image_path, threshold=0.5)
end = time.time()
print("only predict : ", end - start)
labels = [
f"{COCO_CLASSES[class_id]} {confidence:.2f}"
for class_id, confidence in zip(detections.class_id, detections.confidence)
]
image = image.convert("RGB")
annotated_image = sv.BoxAnnotator().annotate(image.copy(), detections)
annotated_image = sv.LabelAnnotator().annotate(annotated_image, detections, labels)
sv.plot_image(annotated_image)
cv2.imwrite(im_save, np.array(annotated_image))
image_url = "path/to/image"
start = time.time()
model = RFDETRLarge()
predict(model, image_url, "predict_old.jpg")
end = time.time()
print("old: ", end - start)
model = RFDETRLarge(onnx_path="output/inference_model.onnx")
predict(model, image_url, "predict_onnx.jpg")
end = time.time()
print("onnx: ", end - start)
The bboxes and confidence scores of the image i tested are the same.
The time using GPU is 0,14 sec for cuda with pth model and 0,11 sec for onnx model
Any specific deployment considerations
Docs
- [ ] Docs updated? What were the changes:
Let's wait this PR until we'll merge https://github.com/roboflow/rf-detr/pull/129.
@valavanisleonidas #129 got merged. We can work on this PR now.
Yes I saw that thank you. should we also have dynamic batch size for onnx models since the predict function support batch inference ?
@SkalskiP I think the PR is mostly ready. Two things to mention.
- in the providers for onnx we can add a device id for the GPU to use (and add an argument as well).
- I tried to export dynamic onnx model but something crashes. Even though the model input is
['batch', 3, 560, 560]it crashes with error in layertransformer/Reshape_13, so probably model does not support dynamic batching in onnx. We have to handle this somehow in predict or maybe fix the model to support dynamic batching ?? :D
Hi, dear developer, how is going on ? I also need to use the ONNX model to predict.
Hi, dear developer, how is going on ? I also need to use the ONNX model to predict.
this PR is not up to date with the latest changes. I can do it if @SkalskiP and the team is still interested in merging it.
Hello @valavanisleonidas any update on this PR? It's a super interesting PR which alot of people are waiting for I guess.
Hello @MustafaKiwaPI ,
it’s not up to me to merge this. In order to use it because I wanted it as well I forked the code with the change and use the forked version.
If the maintainers of the project are still interested we can update the branch in order to merge this.
Hello @valavanisleonidas any update on this PR? It's a super interesting PR which alot of people are waiting for I guess.