DCHECK(mem_span <= data_->size()) failed. mem_span: 2559986, data_->size()320000

[lix19937]

Description

[09/19/2024-13:28:56] [V] [TRT] --------------- Timing Runner: /img_backbone/layer2/layer2.2/conv1/Conv + /img_backbone/layer2/layer2.2/relu/Relu (CublasConvolution)
[09/19/2024-13:28:56] [V] [TRT] CublasConvolution has no valid tactics for this config, skipping
[09/19/2024-13:28:56] [V] [TRT] =============== Computing costs for 
[09/19/2024-13:28:56] [V] [TRT] *************** Autotuning format combination: Half(30,5,1), Half(16,1:8,4,1), Half(16,1:8,4,1), Half(16,1:8,4,1), Half(1280000,1:8,160000,5000), Half(10240000,256,1) -> Half(10240000,256,1), Half(10240000,256,1), Half(10240000,256,1), Half(160000,4,1:8,4,1), Half(80000,2,2,1:8,2,1), Half(2560000,64,8,1:8,2,1), Half(320000,8,8,8,1:8,2,1), Half(320000,8,8,8,1:8,2,1), Half(320000,8,8,8,1:8,2,1), Half(320000,8,8,8,1:8,2,1), Half(320000,8,8,8,1:8,2,1), Half(320000,8,8,8,1:8,2,1), Half(320000,8,8,8,1:8,2,1), Half(320000,8,8,8,1:8,2,1), Half(320000,8,8,8,1:8,2,1), Half(320000,8,8,8,1:8,2,1), Half(320000,8,8,8,1:8,2,1), Half(10240000,256,1), Half(10240000,256,1), Half(5120000,128,16,1:8,2,1), Half(40000,1:8,1,1), Half(40000,1:8,1,1), Half(40000,1:8,1,1), Half(40000,1:8,1,1), Half(40000,1:8,1,1), Half(40000,1:8,1,1), Half(40000,1:8,1,1), Half(40000,1:8,1,1), Half(40000,1:8,1,1), Half(40000,1:8,1,1), Half(40000,1:8,1,1), Half(40000,1:8,1,1), Half(40000,1,1), Half(40000,1,1), Half(40000,1,1), Half(40000,1,1), Half(40000,1,1), Half(40000,1,1), Half(40000,1,1), Half(40000,1,1), Half(40000,1,1), Half(40000,1,1), Half(40000,1,1), Half(40000,1,1) ***************
[09/19/2024-13:28:56] [V] [TRT] --------------- Timing Runner: {ForeignNode[onnx::Expand_1636.../pts_bbox_head/encoder/layers.0/attentions.1/Cast_3]} (Myelin)
operand.cpp:61: DCHECK(mem_span <= data_->size()) failed. mem_span: 2559986, data_->size()320000
Segmentation fault (core dumped)   

./trtexec --onnx=./trt_bev/bevf-864-1536-200-200_trt_plugin_fixed_20240919_prev_bev_opt_15.onnx   \
--fp16   --dumpProfile  \
--saveEngine=./trt8510_fp16.plan \
--tacticSources=-CUDNN,-CUBLAS,-CUBLAS_LT \
--workspace=4096 --verbose

Environment

TensorRT Version:8.5.10 Drive os 6060

NVIDIA GPU: Drive agx orin-x

NVIDIA Driver Version:

CUDA Version:11.4

CUDNN Version:11.6

I think this is bug of trt8.5.10 (limit with drive os version).
@ttyio Can you help confirm it ?

fp32 passsed. fp16 wrong.

[moraxu]

I will follow up with people from Drive OS once they're back next week.

[lix19937]

Anything update ? It related with memory of gpu ?

[moraxu]

Apologies for the delay, I was told to instance an internal bug for this to reproduce. Can you share ./trt_bev/bevf-864-1536-200-200_trt_plugin_fixed_20240919_prev_bev_opt_15.onnx ?