rf-detr
rf-detr copied to clipboard
roboflow
optimize_for_inference RuntimeError
Search before asking
- [x] I have searched the RF-DETR issues and found no similar bug report.
Bug
After updating from rfdetr 1.2.1 to 1.3 I started to get this error for optimize_for_inference:
% uv run python scripts/setup-checks/rfdetr_check.py
rf-detr-nano.pth: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 349M/349M [00:14<00:00, 26.1MiB/s]
Using a different number of positional encodings than DINOv2, which means we're not loading DINOv2 backbone weights. This is not a problem if finetuning a pretrained RF-DETR model.
Using patch size 16 instead of 14, which means we're not loading DINOv2 backbone weights. This is not a problem if finetuning a pretrained RF-DETR model.
Loading pretrain weights
`loss_type=None` was set in the config but it is unrecognized. Using the default loss: `ForCausalLMLoss`.
TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
TracerWarning: torch.as_tensor results are registered as constants in the trace. You can safely ignore this warning if you use this function to create tensors out of constant variables that would be the same every time you call this function. In any other case, this might cause the trace to be incorrect.
TracerWarning: Iterating over a tensor might cause the trace to be incorrect. Passing a tensor of different shape won't change the number of iterations executed (and might lead to errors or silently give incorrect results).
TracerWarning: torch.tensor results are registered as constants in the trace. You can safely ignore this warning if you use this function to create tensors out of constant variables that would be the same every time you call this function. In any other case, this might cause the trace to be incorrect.
TracerWarning: torch.tensor results are registered as constants in the trace. You can safely ignore this warning if you use this function to create tensors out of constant variables that would be the same every time you call this function. In any other case, this might cause the trace to be incorrect.
UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at /Users/runner/work/pytorch/pytorch/pytorch/aten/src/ATen/native/TensorShape.cpp:4324.)
TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
TracerWarning: Iterating over a tensor might cause the trace to be incorrect. Passing a tensor of different shape won't change the number of iterations executed (and might lead to errors or silently give incorrect results).
TracerWarning: Iterating over a tensor might cause the trace to be incorrect. Passing a tensor of different shape won't change the number of iterations executed (and might lead to errors or silently give incorrect results).
Traceback (most recent call last):
File "/Users/cdeil/code/stocadro/stocadro-ai/scripts/setup-checks/rfdetr_check.py", line 4, in <module>
model.optimize_for_inference()
~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
File "/Users/cdeil/code/stocadro/stocadro-ai/.venv/lib/python3.13/site-packages/rfdetr/detr.py", line 99, in optimize_for_inference
self.model.inference_model = torch.jit.trace(
~~~~~~~~~~~~~~~^
self.model.inference_model,
^^^^^^^^^^^^^^^^^^^^^^^^^^^
...<4 lines>...
)
^
)
^
File "/Users/cdeil/code/stocadro/stocadro-ai/.venv/lib/python3.13/site-packages/torch/jit/_trace.py", line 1002, in trace
traced_func = _trace_impl(
func,
...<10 lines>...
_store_inputs,
)
File "/Users/cdeil/code/stocadro/stocadro-ai/.venv/lib/python3.13/site-packages/torch/jit/_trace.py", line 696, in _trace_impl
return trace_module(
func,
...<9 lines>...
_store_inputs=_store_inputs,
)
File "/Users/cdeil/code/stocadro/stocadro-ai/.venv/lib/python3.13/site-packages/torch/jit/_trace.py", line 1282, in trace_module
module._c._create_method_from_trace(
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
method_name,
^^^^^^^^^^^^
...<6 lines>...
_store_inputs,
^^^^^^^^^^^^^^
)
^
RuntimeError: Only tensors, lists, tuples of tensors, or dictionary of tensors can be output from traced functions
Environment
- rfdetr==1.3.0
- MacOS 26
- Python 3.13
- torch 2.8.0
- torchvision 0.23.0
Minimal Reproducible Example
import rfdetr
model = rfdetr.RFDETRNano()
model.optimize_for_inference()
Additional
No response
Are you willing to submit a PR?
- [ ] Yes, I'd like to help by submitting a PR!
I can confirm this issue with 1.3.0
Environment
- rfdetr 1.3.0
- torch 2.8.0+cu129
- torchvision 0.23.0+cu129
- python 3.11.9
- Windows 11
replicated. we only tested the new segmentation model with optimize_for_inference for this release, that works but it appears the object detector is broken. fixing. thanks for the flag
Using torch 2.8.0 and 2.9.1, and trying to run the example on the home page results in the same error:
import io
import requests
import supervision as sv
from PIL import Image
from rfdetr import RFDETRBase
from rfdetr.util.coco_classes import COCO_CLASSES
model = RFDETRBase()
model.optimize_for_inference()
url = "https://media.roboflow.com/notebooks/examples/dog-2.jpeg"
image = Image.open(io.BytesIO(requests.get(url).content))
detections = model.predict(image, threshold=0.5)
labels = [
f"{COCO_CLASSES[class_id]} {confidence:.2f}"
for class_id, confidence in zip(detections.class_id, detections.confidence)
]
annotated_image = image.copy()
annotated_image = sv.BoxAnnotator().annotate(annotated_image, detections)
annotated_image = sv.LabelAnnotator().annotate(annotated_image, detections, labels)
sv.plot_image(annotated_image)
Loading pretrain weights
`loss_type=None` was set in the config but it is unrecognized. Using the default loss: `ForCausalLMLoss`.
TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
TracerWarning: torch.as_tensor results are registered as constants in the trace. You can safely ignore this warning if you use this function to create tensors out of constant variables that would be the same every time you call this function. In any other case, this might cause the trace to be incorrect.
TracerWarning: Iterating over a tensor might cause the trace to be incorrect. Passing a tensor of different shape won't change the number of iterations executed (and might lead to errors or silently give incorrect results).
TracerWarning: torch.tensor results are registered as constants in the trace. You can safely ignore this warning if you use this function to create tensors out of constant variables that would be the same every time you call this function. In any other case, this might cause the trace to be incorrect.
TracerWarning: torch.tensor results are registered as constants in the trace. You can safely ignore this warning if you use this function to create tensors out of constant variables that would be the same every time you call this function. In any other case, this might cause the trace to be incorrect.
UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at /pytorch/aten/src/ATen/native/TensorShape.cpp:4317.)
TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
TracerWarning: Iterating over a tensor might cause the trace to be incorrect. Passing a tensor of different shape won't change the number of iterations executed (and might lead to errors or silently give incorrect results).
TracerWarning: Iterating over a tensor might cause the trace to be incorrect. Passing a tensor of different shape won't change the number of iterations executed (and might lead to errors or silently give incorrect results).
---------------------------------------------------------------------------
RuntimeError Traceback (most recent call last)
Cell In[1], line 10
6 from rfdetr.util.coco_classes import COCO_CLASSES
8 model = RFDETRBase()
---> 10 model.optimize_for_inference()
12 url = "https://media.roboflow.com/notebooks/examples/dog-2.jpeg"
14 image = Image.open(io.BytesIO(requests.get(url).content))
File ~/Work/caml/.venv/lib/python3.13/site-packages/rfdetr/detr.py:99, in RFDETR.optimize_for_inference(self, compile, batch_size, dtype)
96 self._optimized_dtype = dtype
98 if compile:
---> 99 self.model.inference_model = torch.jit.trace(
100 self.model.inference_model,
101 torch.randn(
102 batch_size, 3, self.model.resolution, self.model.resolution,
103 device=self.model.device,
104 dtype=dtype
105 )
106 )
107 self._optimized_has_been_compiled = True
108 self._optimized_batch_size = batch_size
File ~/Work/caml/.venv/lib/python3.13/site-packages/torch/jit/_trace.py:998, in trace(func, example_inputs, optimize, check_trace, check_inputs, check_tolerance, strict, _force_outplace, _module_class, _compilation_unit, example_kwarg_inputs, _store_inputs)
989 warnings.warn(
990 "`optimize` is deprecated and has no effect. "
991 "Use `with torch.jit.optimized_execution()` instead",
992 FutureWarning,
993 stacklevel=2,
994 )
996 from torch._utils_internal import log_torchscript_usage
--> 998 traced_func = _trace_impl(
999 func,
1000 example_inputs,
1001 optimize,
1002 check_trace,
1003 check_inputs,
1004 check_tolerance,
1005 strict,
1006 _force_outplace,
1007 _module_class,
1008 _compilation_unit,
1009 example_kwarg_inputs,
1010 _store_inputs,
1011 )
1012 log_torchscript_usage("trace", model_id=_get_model_id(traced_func))
1013 return traced_func
File ~/Work/caml/.venv/lib/python3.13/site-packages/torch/jit/_trace.py:696, in _trace_impl(func, example_inputs, optimize, check_trace, check_inputs, check_tolerance, strict, _force_outplace, _module_class, _compilation_unit, example_kwarg_inputs, _store_inputs)
694 else:
695 raise RuntimeError("example_kwarg_inputs should be a dict")
--> 696 return trace_module(
697 func,
698 {"forward": example_inputs},
699 None,
700 check_trace,
701 wrap_check_inputs(check_inputs),
702 check_tolerance,
703 strict,
704 _force_outplace,
705 _module_class,
706 example_inputs_is_kwarg=isinstance(example_kwarg_inputs, dict),
707 _store_inputs=_store_inputs,
708 )
709 if (
710 hasattr(func, "__self__")
711 and isinstance(func.__self__, torch.nn.Module)
712 and func.__name__ == "forward"
713 ):
714 if example_inputs is None:
File ~/Work/caml/.venv/lib/python3.13/site-packages/torch/jit/_trace.py:1181, in trace_module(mod, inputs, optimize, check_trace, check_inputs, check_tolerance, strict, _force_outplace, _module_class, _compilation_unit, example_inputs_is_kwarg, _store_inputs)
1179 else:
1180 example_inputs = make_tuple(example_inputs)
-> 1181 module._c._create_method_from_trace(
1182 method_name,
1183 func,
1184 example_inputs,
1185 var_lookup_fn,
1186 strict,
1187 _force_outplace,
1188 argument_names,
1189 _store_inputs,
1190 )
1192 check_trace_method = module._c._get_method(method_name)
1194 # Check the trace against new traces created from user-specified inputs
RuntimeError: Only tensors, lists, tuples of tensors, or dictionary of tensors can be output from traced functions
model.optimize_for_inference(compile=False) is the only workaround I have found currently.
With rf-detr version 1.3.0 I still get the same issue. However @vmiller987 proposed workaround with compile=False works.
Additional environment info: Python 3.12 torch 2.5.1+cu121 roboflow 1.2.11
Only tensors, lists, tuples of tensors, or dictionary of tensors can be output from traced functions
Should work in most recent branch, don't think the fix has made it into a release yet
With rf-detr version 1.3.0 I still get the same issue. However @vmiller987 proposed workaround with compile=False works.
Additional environment info: Python 3.12 torch 2.5.1+cu121 roboflow 1.2.11
Only tensors, lists, tuples of tensors, or dictionary of tensors can be output from traced functions
@DaniVLA I came across this same error and resolved with the temporary model.optimize_for_inference(compile=False) on default branch
Python 3.11.11 torch 2.9.0 + cuda 12.2 roboflow 1.2.11
Should work in most recent branch, don't think the fix has made it into a release yet
I have confirmed master works.
uv venv --python 3.13
source .venv/bin/activate
git clone https://github.com/roboflow/rf-detr.git
uv pip install -e .
import io
import requests
import supervision as sv
from PIL import Image
from rfdetr import RFDETRBase
from rfdetr.util.coco_classes import COCO_CLASSES
model = RFDETRBase()
model.optimize_for_inference()
url = "https://media.roboflow.com/notebooks/examples/dog-2.jpeg"
image = Image.open(io.BytesIO(requests.get(url).content))
detections = model.predict(image, threshold=0.5)
labels = [
f"{COCO_CLASSES[class_id]} {confidence:.2f}"
for class_id, confidence in zip(detections.class_id, detections.confidence)
]
annotated_image = image.copy()
annotated_image = sv.BoxAnnotator().annotate(annotated_image, detections)
annotated_image = sv.LabelAnnotator().annotate(annotated_image, detections, labels)
sv.plot_image(annotated_image)
SyntaxWarning: invalid escape sequence '\s'
SyntaxWarning: invalid escape sequence '\s'
rf-detr-base.pth: 100%|██████████| 355M/355M [00:05<00:00, 68.7MiB/s]
Loading pretrain weights
`loss_type=None` was set in the config but it is unrecognized. Using the default loss: `ForCausalLMLoss`.
TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
TracerWarning: torch.as_tensor results are registered as constants in the trace. You can safely ignore this warning if you use this function to create tensors out of constant variables that would be the same every time you call this function. In any other case, this might cause the trace to be incorrect.
TracerWarning: Iterating over a tensor might cause the trace to be incorrect. Passing a tensor of different shape won't change the number of iterations executed (and might lead to errors or silently give incorrect results).
TracerWarning: torch.tensor results are registered as constants in the trace. You can safely ignore this warning if you use this function to create tensors out of constant variables that would be the same every time you call this function. In any other case, this might cause the trace to be incorrect.
TracerWarning: torch.tensor results are registered as constants in the trace. You can safely ignore this warning if you use this function to create tensors out of constant variables that would be the same every time you call this function. In any other case, this might cause the trace to be incorrect.
UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at /pytorch/aten/src/ATen/native/TensorShape.cpp:4317.)
TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
TracerWarning: Iterating over a tensor might cause the trace to be incorrect. Passing a tensor of different shape won't change the number of iterations executed (and might lead to errors or silently give incorrect results).
TracerWarning: Iterating over a tensor might cause the trace to be incorrect. Passing a tensor of different shape won't change the number of iterations executed (and might lead to errors or silently give incorrect results).