convert onnx to trt, onnxparser fail, but trtexec success

[FreshZZ]

Description

convert onnx to trt, use trtexec success, but onnxparser fail, error is : [06/23/2022-13:33:48] [E] [TRT] 10: [optimizer.cpp::computeCosts::1855] Error Code 10: Internal Error (Could not find any implementation for node TopK_531.)

the code use onnxparser is below, could you tell me where I did wrong: ` if (!parser->parseFromFile(cfg.onnx.c_str(), static_cast(sample::gLogger.getReportableSeverity()))){ std::cerr << "Parsing File Failed" << std::endl; return -1; } config->setMaxWorkspaceSize(cfg.workspace); config -> setFlag(nvinfer1::BuilderFlag::kFP16);

auto profile = builder->createOptimizationProfile(); for(auto idx = cfg.minShapes.begin(); idx!=cfg.minShapes.end(); ++idx){ string input_name = (*idx).first; if (cfg.maxShapes.count(input_name)==0 || cfg.optShapes.count(input_name)==0){ std::cerr<<"dyn shape need set minShapes, optShapes, maxShapes"<<std::endl; return -1; } profile->setDimensions(input_name.c_str(), nvinfer1::OptProfileSelector::kMIN, nvinfer1::Dims4{cfg.minShapes[input_name][0], cfg.minShapes[input_name][1], cfg.minShapes[input_name][2], cfg.minShapes[input_name][3]}); profile->setDimensions(input_name.c_str(), nvinfer1::OptProfileSelector::kOPT, nvinfer1::Dims4{cfg.optShapes[input_name][0], cfg.optShapes[input_name][1], cfg.optShapes[input_name][2], cfg.optShapes[input_name][3]}); profile->setDimensions(input_name.c_str(), nvinfer1::OptProfileSelector::kMAX, nvinfer1::Dims4{cfg.maxShapes[input_name][0], cfg.maxShapes[input_name][1], cfg.maxShapes[input_name][2], cfg.maxShapes[input_name][3]}); } config->addOptimizationProfile(profile); auto CudaEngine = std::shared_ptrnvinfer1::ICudaEngine(builder->buildEngineWithConfig(*network, *config), InferDeleter());// got error here `

Environment

TensorRT Version: 8.0.0.3 NVIDIA GPU: T4 NVIDIA Driver Version: 460.32.03 CUDA Version: 11.2 CUDNN Version: 8.0 Operating System: Ubuntu 20.04.3 LTS Python Version (if applicable): Tensorflow Version (if applicable): PyTorch Version (if applicable): Baremetal or Container (if so, version):

Relevant Files

Steps To Reproduce

[zerollzeng]

Can you provide a reproduce sample here? also it would be good to try our latest release.

[FreshZZ]

@zerollzeng thanks for your reply， reproduce sample : debug model , the docker image I used is nvcr.io/nvidia/tensorrt:21.10-py3

[nvpohanh]

@FreshZZ If trtexec works but not your script, it usually means that there are some incorrect API usage in your script. Could you share your script that uses onnxparser? Specifically, I would like to know how you created the network. Thanks

[FreshZZ]

@nvpohanh the full script is onnxparser the model is maskrcnn, onnx file and tensorrt plugins obtained from mmdeploy

[zanazakaryaie]

I have the same problem. trtexec can parse my onnx file with this log:

[01/15/2024-16:38:03] [I] Start parsing network model.
[01/15/2024-16:38:03] [I] [TRT] ----------------------------------------------------------------
[01/15/2024-16:38:03] [I] [TRT] Input filename:   model.onnx
[01/15/2024-16:38:03] [I] [TRT] ONNX IR version:  0.0.6
[01/15/2024-16:38:03] [I] [TRT] Opset version:    11
[01/15/2024-16:38:03] [I] [TRT] Producer name:    pytorch
[01/15/2024-16:38:03] [I] [TRT] Producer version: 2.0.1
[01/15/2024-16:38:03] [I] [TRT] Domain:           
[01/15/2024-16:38:03] [I] [TRT] Model version:    0
[01/15/2024-16:38:03] [I] [TRT] Doc string:       
[01/15/2024-16:38:03] [I] [TRT] ----------------------------------------------------------------
[01/15/2024-16:38:03] [W] [TRT] onnx2trt_utils.cpp:374: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[01/15/2024-16:38:03] [W] [TRT] onnx2trt_utils.cpp:400: One or more weights outside the range of INT32 was clamped
[01/15/2024-16:38:03] [I] Finished parsing network model. Parse time: 0.0661147

But I get the following error when I use an instance of IParser: Assertion failed: mode != "cubic" && "This version of TensorRT does not support cubic interpolation!"

Here is the code that I use for parsing:

auto builder = std::unique_ptr<nvinfer1::IBuilder>(nvinfer1::createInferBuilder(logger));
if (!builder) {
    throw std::runtime_error("Failed to create IBuilder");
}

const uint32_t flag = 1U << static_cast<uint32_t>(nvinfer1::NetworkDefinitionCreationFlag::kEXPLICIT_BATCH);
auto networkDefinition = std::unique_ptr<nvinfer1::INetworkDefinition>(mBuilder->createNetworkV2(flag));
if (!networkDefinition) {
    throw std::runtime_error("Failed to create INetworkDefinition");
}

auto parser = std::unique_ptr<nvonnxparser::IParser>(nvonnxparser::createParser(*networkDefinition, logger));
if (!parser) {
    throw std::runtime_error("Failed to create Onnx Parser");
}

if (!parser->parseFromFile(filepath.c_str(), static_cast<int>(nvinfer1::ILogger::Severity::kINFO))) {
    for (int32_t i = 0; i < parser->getNbErrors(); ++i) {
        std::cout << parser->getError(i)->desc()) << std::endl;
    }
    throw std::runtime_error("Failed to parse onnx file");
}

[nvpohanh]

cubic mode support was not added until TRT 8.5

[zanazakaryaie]

@nvpohanh My TRT version is 8.6.1 I also added initLibNvInferPlugins before calling any TRT C++ API call. But still getting the error

[nvpohanh]

@zanazakaryaie Which container did you use? How did you install TRT 8.6? Did you check why your parser code doesn't visit this part of TRT 8.6 ONNX parser code path, which already supports "cubic": https://github.com/onnx/onnx-tensorrt/blob/release/8.6-GA/builtin_op_importers.cpp#L3746-L3749

The assert you shown was here in TRT 8.4: https://github.com/onnx/onnx-tensorrt/blob/release/8.4-GA/builtin_op_importers.cpp#L3419-L3420

[zanazakaryaie]

I'm using a container that has TRT 8.6. This is the output of dpkg -l | grep nvinf:

ii  libnvinfer-bin                         8.6.1.6-1+cuda11.8                 amd64        TensorRT binaries
ii  libnvinfer-dev                         8.6.1.6-1+cuda11.8                 amd64        TensorRT development libraries
ii  libnvinfer-dispatch-dev                8.6.1.6-1+cuda11.8                 amd64        TensorRT development dispatch runtime libraries
ii  libnvinfer-dispatch8                   8.6.1.6-1+cuda11.8                 amd64        TensorRT dispatch runtime library
ii  libnvinfer-headers-dev                 8.6.1.6-1+cuda11.8                 amd64        TensorRT development headers
ii  libnvinfer-headers-plugin-dev          8.6.1.6-1+cuda11.8                 amd64        TensorRT plugin headers
ii  libnvinfer-lean-dev                    8.6.1.6-1+cuda11.8                 amd64        TensorRT lean runtime libraries
ii  libnvinfer-lean8                       8.6.1.6-1+cuda11.8                 amd64        TensorRT lean runtime library
ii  libnvinfer-plugin-dev                  8.6.1.6-1+cuda11.8                 amd64        TensorRT plugin libraries
ii  libnvinfer-plugin8                     8.6.1.6-1+cuda11.8                 amd64        TensorRT plugin libraries
ii  libnvinfer-samples                     8.6.1.6-1+cuda11.8                 all          TensorRT samples
ii  libnvinfer-vc-plugin-dev               8.6.1.6-1+cuda11.8                 amd64        TensorRT vc-plugin library
ii  libnvinfer-vc-plugin8                  8.6.1.6-1+cuda11.8                 amd64        TensorRT vc-plugin library
ii  libnvinfer8                            8.6.1.6-1+cuda11.8                 amd64        TensorRT runtime libraries

And this is the output of ldd myExecutable:

linux-vdso.so.1 (0x00007ffc0559e000)
libnvinfer.so.8 => /usr/lib/x86_64-linux-gnu/libnvinfer.so.8 (0x00007fe123b57000)
libnvonnxparser.so.8 => /usr/local/lib/libnvonnxparser.so.8 (0x00007fe123a42000)
libnvinfer_plugin.so.8 => /usr/lib/x86_64-linux-gnu/libnvinfer_plugin.so.8 (0x00007fe121680000)
libdl.so.2 => /usr/lib/x86_64-linux-gnu/libdl.so.2 (0x00007fe12167a000)
librt.so.1 => /usr/lib/x86_64-linux-gnu/librt.so.1 (0x00007fe12166e000)
libpthread.so.0 => /usr/lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fe12164b000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007fe121469000)
libgcc_s.so.1 => /usr/lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fe12144e000)
libc.so.6 => /usr/lib/x86_64-linux-gnu/libc.so.6 (0x00007fe12125c000)
/lib64/ld-linux-x86-64.so.2 (0x00007fe132494000)
libcppfs.so.1 => /usr/local/lib/libcppfs.so.1 (0x00007fe121203000)
libyaml-cpp.so.0.6 => /usr/lib/x86_64-linux-gnu/libyaml-cpp.so.0.6 (0x00007fe1211a6000)
libm.so.6 => /usr/lib/x86_64-linux-gnu/libm.so.6 (0x00007fe121057000)
libprotobuf.so.17 => /usr/lib/x86_64-linux-gnu/libprotobuf.so.17 (0x00007fe120d5e000)
libcublas.so.11 => /usr/local/cuda/targets/x86_64-linux/lib/libcublas.so.11 (0x00007fe11b100000)
libcublasLt.so.11 => /usr/local/cuda/targets/x86_64-linux/lib/libcublasLt.so.11 (0x00007fe0f6b7a000)
libcudnn.so.8 => /usr/lib/x86_64-linux-gnu/libcudnn.so.8 (0x00007fe0f6952000)
libz.so.1 => /usr/lib/x86_64-linux-gnu/libz.so.1 (0x00007fe0f6936000)

[zanazakaryaie]

The weird thing is that trtexec can parse the onnx file but when I use C++ API calls, it can't!

[nvpohanh]

libnvonnxparser.so.8 => /usr/local/lib/libnvonnxparser.so.8 (0x00007fe123a42000)

Could you check if /usr/local/lib/libnvonnxparser.so.8 indeed come from TRT 8.6 package? Thanks

[zanazakaryaie]

@nvpohanh Sorry I forgot to write back here but earlier today I found that for some weird reason, the libnvonnxparser.so.8 in /usr/local/lib actually comes from TRT8.2! I forced my executable to link against version 8.6 of libnvonnxparser and it fixed the issue. Anyway, thanks for your support and feel free to close this ticket if you want to do so Cheers