FastDeploy icon indicating copy to clipboard operation
FastDeploy copied to clipboard

Published 20 hours ago • verified publisher icon PaddlePaddle

使用c++版本的PPOCRV3时，cpu和GPU能正常进行推理，但是基于tensorrt推理出错

Open jasper-cell opened this issue 2 years ago • 4 comments

环境

  • FastDeploy版本： 说明具体的版本，如fastdeploy-linux-x64-gpu-0.8.0.tgz
  • 系统平台: Linux x64(Ubuntu 18.04)
  • 硬件： 说明具体硬件型号，如 Nvidia GPU 3090， CUDA 11.6 CUDNN 8.3
  • 编译语言： C++

问题描述

[INFO] fastdeploy/backends/tensorrt/trt_backend.cc(623)::CreateTrtEngineFromOnnx Detect serialized TensorRT Engine file in ./ch_PP-OCRv3_det_infer/det_trt_cache.trt, will load it directly. [INFO] fastdeploy/backends/tensorrt/trt_backend.cc(108)::LoadTrtCache Build TensorRT Engine from cache file: ./ch_PP-OCRv3_det_infer/det_trt_cache.trt with shape range information as below, [INFO] fastdeploy/backends/tensorrt/trt_backend.cc(111)::LoadTrtCache Input name: x, shape=[-1, 3, -1, -1], min=[1, 3, 50, 50], max=[1, 3, 1536, 1536]

[INFO] fastdeploy/runtime.cc(506)::Init Runtime initialized with Backend::TRT in Device::GPU. [INFO] fastdeploy/backends/tensorrt/trt_backend.cc(623)::CreateTrtEngineFromOnnx Detect serialized TensorRT Engine file in ./ch_ppocr_mobile_v2.0_cls_infer/cls_trt_cache.trt, will load it directly. [WARNING] fastdeploy/backends/tensorrt/utils.cc(40)::Update [New Shape Out of Range] input name: x, shape: [8, 3, 48, 1024], The shape range before: min_shape=[1, 3, 48, 10], max_shape=[1, 3, 48, 1024]. [WARNING] fastdeploy/backends/tensorrt/utils.cc(52)::Update [New Shape Out of Range] The updated shape range now: min_shape=[1, 3, 48, 10], max_shape=[8, 3, 48, 1024]. [INFO] fastdeploy/backends/tensorrt/trt_backend.cc(108)::LoadTrtCache Build TensorRT Engine from cache file: ./ch_ppocr_mobile_v2.0_cls_infer/cls_trt_cache.trt with shape range information as below, [INFO] fastdeploy/backends/tensorrt/trt_backend.cc(111)::LoadTrtCache Input name: x, shape=[-1, 3, -1, -1], min=[1, 3, 48, 10], max=[8, 3, 48, 1024]

[INFO] fastdeploy/runtime.cc(506)::Init Runtime initialized with Backend::TRT in Device::GPU. [INFO] fastdeploy/backends/tensorrt/trt_backend.cc(623)::CreateTrtEngineFromOnnx Detect serialized TensorRT Engine file in ./ch_PP-OCRv3_rec_infer/rec_trt_cache.trt, will load it directly. [INFO] fastdeploy/backends/tensorrt/trt_backend.cc(108)::LoadTrtCache Build TensorRT Engine from cache file: ./ch_PP-OCRv3_rec_infer/rec_trt_cache.trt with shape range information as below, [INFO] fastdeploy/backends/tensorrt/trt_backend.cc(111)::LoadTrtCache Input name: x, shape=[-1, 3, 48, -1], min=[1, 3, 48, 10], max=[1, 3, 48, 2304]

[INFO] fastdeploy/runtime.cc(506)::Init Runtime initialized with Backend::TRT in Device::GPU. [WARNING] fastdeploy/backends/tensorrt/utils.cc(40)::Update [New Shape Out of Range] input name: x, shape: [4, 3, 48, 584], The shape range before: min_shape=[1, 3, 48, 10], max_shape=[1, 3, 48, 2304]. [WARNING] fastdeploy/backends/tensorrt/utils.cc(52)::Update [New Shape Out of Range] The updated shape range now: min_shape=[1, 3, 48, 10], max_shape=[4, 3, 48, 2304]. [WARNING] fastdeploy/backends/tensorrt/trt_backend.cc(291)::Infer TensorRT engine will be rebuilt once shape range information changed, this may take lots of time, you can set a proper shape range before loading model to avoid rebuilding process. refer https://github.com/PaddlePaddle/FastDeploy/blob/develop/docs/en/faq/tensorrt_tricks.md for more details. [INFO] fastdeploy/backends/tensorrt/trt_backend.cc(479)::BuildTrtEngine Start to building TensorRT Engine... [ERROR] fastdeploy/backends/tensorrt/trt_backend.cc(238)::log 10: [optimizer.cpp::computeCosts::2011] Error Code 10: Internal Error (Could not find any implementation for node {ForeignNode[linear_13.w_0 + (Unnamed Layer* 321) [Shuffle]...p2o.Reshape.23 + p2o.Transpose.7]}.) [ERROR] fastdeploy/backends/tensorrt/trt_backend.cc(238)::log 2: [builder.cpp::buildSerializedNetwork::609] Error Code 2: Internal Error (Assertion enginePtr != nullptr failed. ) [ERROR] fastdeploy/backends/tensorrt/trt_backend.cc(542)::BuildTrtEngine Failed to call buildSerializedNetwork(). Segmentation fault (core dumped) Makefile:89: recipe for target 'run' failed make: *** [run] Error 139

jasper-cell avatar Nov 29 '22 12:11 jasper-cell

环境

  • FastDeploy版本： 说明具体的版本，如fastdeploy-linux-x64-gpu-0.8.0.tgz
  • 系统平台: Linux x64(Ubuntu 18.04)
  • 硬件： 说明具体硬件型号，如 Nvidia GPU 3090， CUDA 11.6 CUDNN 8.3
  • 编译语言： C++

问题描述

[INFO] fastdeploy/backends/tensorrt/trt_backend.cc(623)::CreateTrtEngineFromOnnx Detect serialized TensorRT Engine file in ./ch_PP-OCRv3_det_infer/det_trt_cache.trt, will load it directly. [INFO] fastdeploy/backends/tensorrt/trt_backend.cc(108)::LoadTrtCache Build TensorRT Engine from cache file: ./ch_PP-OCRv3_det_infer/det_trt_cache.trt with shape range information as below, [INFO] fastdeploy/backends/tensorrt/trt_backend.cc(111)::LoadTrtCache Input name: x, shape=[-1, 3, -1, -1], min=[1, 3, 50, 50], max=[1, 3, 1536, 1536]

[INFO] fastdeploy/runtime.cc(506)::Init Runtime initialized with Backend::TRT in Device::GPU. [INFO] fastdeploy/backends/tensorrt/trt_backend.cc(623)::CreateTrtEngineFromOnnx Detect serialized TensorRT Engine file in ./ch_ppocr_mobile_v2.0_cls_infer/cls_trt_cache.trt, will load it directly. [WARNING] fastdeploy/backends/tensorrt/utils.cc(40)::Update [New Shape Out of Range] input name: x, shape: [8, 3, 48, 1024], The shape range before: min_shape=[1, 3, 48, 10], max_shape=[1, 3, 48, 1024]. [WARNING] fastdeploy/backends/tensorrt/utils.cc(52)::Update [New Shape Out of Range] The updated shape range now: min_shape=[1, 3, 48, 10], max_shape=[8, 3, 48, 1024]. [INFO] fastdeploy/backends/tensorrt/trt_backend.cc(108)::LoadTrtCache Build TensorRT Engine from cache file: ./ch_ppocr_mobile_v2.0_cls_infer/cls_trt_cache.trt with shape range information as below, [INFO] fastdeploy/backends/tensorrt/trt_backend.cc(111)::LoadTrtCache Input name: x, shape=[-1, 3, -1, -1], min=[1, 3, 48, 10], max=[8, 3, 48, 1024]

[INFO] fastdeploy/runtime.cc(506)::Init Runtime initialized with Backend::TRT in Device::GPU. [INFO] fastdeploy/backends/tensorrt/trt_backend.cc(623)::CreateTrtEngineFromOnnx Detect serialized TensorRT Engine file in ./ch_PP-OCRv3_rec_infer/rec_trt_cache.trt, will load it directly. [INFO] fastdeploy/backends/tensorrt/trt_backend.cc(108)::LoadTrtCache Build TensorRT Engine from cache file: ./ch_PP-OCRv3_rec_infer/rec_trt_cache.trt with shape range information as below, [INFO] fastdeploy/backends/tensorrt/trt_backend.cc(111)::LoadTrtCache Input name: x, shape=[-1, 3, 48, -1], min=[1, 3, 48, 10], max=[1, 3, 48, 2304]

[INFO] fastdeploy/runtime.cc(506)::Init Runtime initialized with Backend::TRT in Device::GPU. [WARNING] fastdeploy/backends/tensorrt/utils.cc(40)::Update [New Shape Out of Range] input name: x, shape: [4, 3, 48, 584], The shape range before: min_shape=[1, 3, 48, 10], max_shape=[1, 3, 48, 2304]. [WARNING] fastdeploy/backends/tensorrt/utils.cc(52)::Update [New Shape Out of Range] The updated shape range now: min_shape=[1, 3, 48, 10], max_shape=[4, 3, 48, 2304]. [WARNING] fastdeploy/backends/tensorrt/trt_backend.cc(291)::Infer TensorRT engine will be rebuilt once shape range information changed, this may take lots of time, you can set a proper shape range before loading model to avoid rebuilding process. refer https://github.com/PaddlePaddle/FastDeploy/blob/develop/docs/en/faq/tensorrt_tricks.md for more details. [INFO] fastdeploy/backends/tensorrt/trt_backend.cc(479)::BuildTrtEngine Start to building TensorRT Engine... [ERROR] fastdeploy/backends/tensorrt/trt_backend.cc(238)::log 10: [optimizer.cpp::computeCosts::2011] Error Code 10: Internal Error (Could not find any implementation for node {ForeignNode[linear_13.w_0 + (Unnamed Layer* 321) [Shuffle]...p2o.Reshape.23 + p2o.Transpose.7]}.) [ERROR] fastdeploy/backends/tensorrt/trt_backend.cc(238)::log 2: [builder.cpp::buildSerializedNetwork::609] Error Code 2: Internal Error (Assertion enginePtr != nullptr failed. ) [ERROR] fastdeploy/backends/tensorrt/trt_backend.cc(542)::BuildTrtEngine Failed to call buildSerializedNetwork(). Segmentation fault (core dumped) Makefile:89: recipe for target 'run' failed make: *** [run] Error 139

Hello你好， 看着像是在rebuild引擎的时候出的错,你的TensorRT版本是多少呢? 你可以尝试按照这个代码中的 https://github.com/PaddlePaddle/FastDeploy/blob/develop/examples/vision/ocr/PP-OCRv3/cpp/infer.cc 动态shape,改一下动态输入, 并且重新编译example，再来运行. 重新跑之前把模型里的trt engine cache删掉.

yunyaoXYY avatar Nov 29 '22 13:11 yunyaoXYY

image 您好，TensorRT的动态库我直接使用的是解压后的库路径

jasper-cell avatar Nov 29 '22 14:11 jasper-cell

image 您好，TensorRT的动态库我直接使用的是解压后的库路径

Hi, 有尝试过修改动态shape吗, 或者直接尝试一下今天新发布的 FastDeploy 1.0 试一下呢

yunyaoXYY avatar Nov 30 '22 11:11 yunyaoXYY

好的，谢谢。1.0版本我测试了，可以正常使用tensorrt了。想请问下，端到端的PGNet有打算出fastDeploy的cpp版本吗

jasper-cell avatar Nov 30 '22 11:11 jasper-cell

环境 FastDeploy版本： fastdeploy-1.0.6.tgz 系统平台: Linux x64(Ubuntu 18.04) 硬件：jetpack4.6.3， CUDA 10.2 CUDNN 8.2.1 tensorrt8.2 编译语言： C++

问题描述 ./infer_demo ../model/ch_PP-OCRv3_det_infer ../model/ch_ppocr_mobile_v2.0_cls_infer ../model/ch_PP-OCRv3_rec_infer ../model/ppocr_keys_v1.txt ../data/12.jpg 7 [WARNING] fastdeploy/runtime/runtime_option.cc(376)::SetTrtInputShape RuntimeOption::SetTrtInputShape will be removed in v1.2.0, please use RuntimeOption.trt_option.SetShape() instead. [WARNING] fastdeploy/runtime/runtime_option.cc(376)::SetTrtInputShape RuntimeOption::SetTrtInputShape will be removed in v1.2.0, please use RuntimeOption.trt_option.SetShape() instead. [WARNING] fastdeploy/runtime/runtime_option.cc(376)::SetTrtInputShape RuntimeOption::SetTrtInputShape will be removed in v1.2.0, please use RuntimeOption.trt_option.SetShape() instead. [INFO] fastdeploy/runtime/backends/tensorrt/trt_backend.cc(556)::BuildTrtEngineStart to building TensorRT Engine... [INFO] fastdeploy/runtime/backends/tensorrt/trt_backend.cc(643)::BuildTrtEngineTensorRT Engine is built successfully. [INFO] fastdeploy/runtime/runtime.cc(306)::CreateTrtBackend Runtime initialized with Backend::TRT in Device::GPU. [INFO] fastdeploy/runtime/backends/tensorrt/trt_backend.cc(556)::BuildTrtEngineStart to building TensorRT Engine... [INFO] fastdeploy/runtime/backends/tensorrt/trt_backend.cc(643)::BuildTrtEngineTensorRT Engine is built successfully. [INFO] fastdeploy/runtime/runtime.cc(306)::CreateTrtBackend Runtime initialized with Backend::TRT in Device::GPU. [INFO] fastdeploy/runtime/backends/tensorrt/trt_backend.cc(556)::BuildTrtEngineStart to building TensorRT Engine... [ERROR] fastdeploy/runtime/backends/tensorrt/trt_backend.cc(239)::log 10: [optimizer.cpp::computeCosts::2011] Error Code 10: Internal Error (Could not find any implementation for node {ForeignNode[linear_13.w_0 + (Unnamed Layer* 321) [Shuffle]...p2o.Reshape.23 + p2o.Transpose.7]}.) [ERROR] fastdeploy/runtime/backends/tensorrt/trt_backend.cc(239)::log 2: [builder.cpp::buildSerializedNetwork::609] Error Code 2: Internal Error (Assertion enginePtr != nullptr failed. ) [ERROR] fastdeploy/runtime/backends/tensorrt/trt_backend.cc(620)::BuildTrtEngineFailed to call buildSerializedNetwork(). [ERROR] fastdeploy/runtime/backends/tensorrt/trt_backend.cc(736)::CreateTrtEngineFromOnnx Failed to build tensorrt engine. [INFO] fastdeploy/runtime/runtime.cc(306)::CreateTrtBackend Runtime initialized with Backend::TRT in Device::GPU. [ERROR] fastdeploy/runtime/backends/tensorrt/trt_backend.cc(447)::SetInputs TRTBackend SetInputs not find name:x Aborted (core dumped)

说明 使用c++版本的PPOCRV3时，CPU和GPU GPU-TRT能正常进行推理，但是使用Nvidia TensorRT推理出错

catofyuanyuan avatar May 17 '23 09:05 catofyuanyuan