DS4SD
Unable to run inference on GPU
Bug
return forward_call(*args, **kwargs) RuntimeError: The following operation failed in the TorchScript interpreter. Traceback of TorchScript, serialized code (most recent call last): File "code/torch.py", line 13, in forward postprocessor = self.postprocessor model = self.model _0, _1, = (model).forward(images, ) ~~~~~~~~~~~~~~ <--- HERE _2 = (postprocessor).forward(_0, orig_target_sizes, _1, ) _3, _4, _5, = _2 File "code/torch/src/zoo/rtdetr/rtdetr.py", line 15, in forward backbone = self.backbone _0, _1, _2, = (backbone).forward(images, ) _3, _4, _5, = (encoder).forward(_0, _1, _2, ) ~~~~~~~~~~~~~~~~ <--- HERE _6, _7, = (decoder).forward(_3, _4, _5, ) return (_6, _7) File "code/torch/src/zoo/rtdetr/hybrid_encoder.py", line 66, in forward _24 = torch.slice(_23, 2, 0, 9223372036854775807) pos_embed = torch.to(_24, dtype=6, layout=0, device=torch.device("cpu")) _25 = torch.permute((_03).forward(tensor, pos_embed, ), [0, 2, 1]) ~~~~~~~~~~~~ <--- HERE _26 = torch.reshape(_25, [-1, 256, _3, _7]) input = torch.contiguous(_26) File "code/torch/src/zoo/rtdetr/hybrid_encoder.py", line 91, in forward layers = self.layers _0 = getattr(layers, "0") return (_0).forward(tensor, pos_embed, ) ~~~~~~~~~~~ <--- HERE class TransformerEncoderLayer(Module): parameters = [] File "code/torch/src/zoo/rtdetr/hybrid_encoder.py", line 118, in forward dropout1 = self.dropout1 self_attn = self.self_attn x = torch.add(tensor, pos_embed) ~~~~~~~~~ <--- HERE _24 = (dropout1).forward((self_attn).forward(x, tensor, ), ) input = torch.add(tensor, _24)
Traceback of TorchScript, original code (most recent call last):
/data/ahn/gits/RT-DETR/rtdetr_pytorch/tools/../src/zoo/rtdetr/hybrid_encoder.py(141): with_pos_embed
/data/ahn/gits/RT-DETR/rtdetr_pytorch/tools/../src/zoo/rtdetr/hybrid_encoder.py(147): forward
/data/ahn/gits/RT-DETR/rtdetr_pytorch/pytorch_env/lib/python3.9/site-packages/torch/nn/modules/module.py(1488): _slow_forward
/data/ahn/gits/RT-DETR/rtdetr_pytorch/pytorch_env/lib/python3.9/site-packages/torch/nn/modules/module.py(1501): _call_impl
/data/ahn/gits/RT-DETR/rtdetr_pytorch/tools/../src/zoo/rtdetr/hybrid_encoder.py(174): forward
/data/ahn/gits/RT-DETR/rtdetr_pytorch/pytorch_env/lib/python3.9/site-packages/torch/nn/modules/module.py(1488): _slow_forward
/data/ahn/gits/RT-DETR/rtdetr_pytorch/pytorch_env/lib/python3.9/site-packages/torch/nn/modules/module.py(1501): _call_impl
/data/ahn/gits/RT-DETR/rtdetr_pytorch/tools/../src/zoo/rtdetr/hybrid_encoder.py(299): forward
/data/ahn/gits/RT-DETR/rtdetr_pytorch/pytorch_env/lib/python3.9/site-packages/torch/nn/modules/module.py(1488): _slow_forward
/data/ahn/gits/RT-DETR/rtdetr_pytorch/pytorch_env/lib/python3.9/site-packages/torch/nn/modules/module.py(1501): _call_impl
/data/ahn/gits/RT-DETR/rtdetr_pytorch/tools/../src/zoo/rtdetr/rtdetr.py(34): forward
/data/ahn/gits/RT-DETR/rtdetr_pytorch/pytorch_env/lib/python3.9/site-packages/torch/nn/modules/module.py(1488): _slow_forward
/data/ahn/gits/RT-DETR/rtdetr_pytorch/pytorch_env/lib/python3.9/site-packages/torch/nn/modules/module.py(1501): _call_impl
/data/ahn/gits/RT-DETR/rtdetr_pytorch/tools/export_torchscript.py(43): forward
/data/ahn/gits/RT-DETR/rtdetr_pytorch/pytorch_env/lib/python3.9/site-packages/torch/nn/modules/module.py(1488): _slow_forward
/data/ahn/gits/RT-DETR/rtdetr_pytorch/pytorch_env/lib/python3.9/site-packages/torch/nn/modules/module.py(1501): _call_impl
/data/ahn/gits/RT-DETR/rtdetr_pytorch/pytorch_env/lib/python3.9/site-packages/torch/jit/_trace.py(1056): trace_module
/data/ahn/gits/RT-DETR/rtdetr_pytorch/pytorch_env/lib/python3.9/site-packages/torch/jit/_trace.py(794): trace
/data/ahn/gits/RT-DETR/rtdetr_pytorch/tools/export_torchscript.py(54): main
/data/ahn/gits/RT-DETR/rtdetr_pytorch/tools/export_torchscript.py(76):
Steps to reproduce
...
Docling version
docling 2.4.0 ...
Python version
3.10.0 ... I'm not able to run inference on GPU, please let me know if there is anything I can do. Thanks in advance!
@poojitha0892 This is unfortunately a known issue with our layout model. We are working on addressing this.
See also: https://github.com/DS4SD/docling/issues/308
