onnx model convert by tf2onnx can't work in c++

[zx-lhb]

Describe the bug

hi, I used the tool successfully converted the saved_model to onnx model, but when I used the onnx model to do inference in cpp, it can't work, the program just exit with exception. When I test with onnx model converted by torch, it worked fine. Could you give any suggestion？Thank you! This is the program cout msg:

2023-05-18 14:21:33.2750436 [E:onnxruntime:, sequential_executor.cc:346 onnxruntime::SequentialExecutor::Execute] Non-zero status code returned while running Transpose node. Name:'StatefulPartitionedCall/attnGateVnet3d/conv3d_48/Conv3D__890' Status Message: CUDA error cudaErrorInvalidConfiguration:invalid configuration argument

and the c++ code:

#include <onnxruntime_cxx_api.h>
#include <onnxruntime_c_api.h>
#include <tensorrt_provider_factory.h>

#include<iostream>
using namespace std;

int main()
{
    
    wstring model_path = L"in272_20230518.onnx";
    
    Ort::Env env(ORT_LOGGING_LEVEL_WARNING, "airway");
    Ort::SessionOptions session_options;
  
    OrtCUDAProviderOptions cuda_options{};
    session_options.AppendExecutionProvider_CUDA(cuda_options);

    Ort::AllocatorWithDefaultOptions allocator;   

    
    Ort::Session session(env, model_path.c_str(), session_options);
    

    Ort::MemoryInfo memory_info = Ort::MemoryInfo::CreateCpu(OrtAllocatorType::OrtArenaAllocator, OrtMemType::OrtMemTypeDefault);

    

    std::vector<const char*> input_node_names = { "input_1"};
    std::vector<const char*> output_node_names = { "pred" };

    std::vector<float> input_image_1(272*272*32,0);
    float* input_1 = input_image_1.data();
    

    cout << session.GetInputTypeInfo(0).GetTensorTypeAndShapeInfo().GetElementType() << endl;;
    cout << input_image_1.size() << endl;

    std::vector<int64_t> input_shape{ 1,272,272,32,1 };
    std::vector<Ort::Value> input_tensors;
    input_tensors.push_back(
        Ort::Value::CreateTensor<float>(memory_info, input_1, 
            input_image_1.size(), input_shape.data(), input_shape.size()));

    std::vector<Ort::Value> output_tensors;

    output_tensors = session.Run(
        Ort::RunOptions{ nullptr },
        input_node_names.data(), //
        input_tensors.data(),     //input tensors
        input_tensors.size(),     //1
        output_node_names.data(), //
        output_node_names.size()); //1

	return 0;
}

Urgency

System information

• OS Platform and Distribution (e.g., Linux Ubuntu 18.04*): windows
• TensorFlow Version: tf2.10
• Python version: py3.9
• ONNX version (if applicable, e.g. 1.11*): 1.13
• ONNXRuntime version (if applicable, e.g. 1.11*): 1.14
• ONNXRuntime-cpp-windows: 1.9

To Reproduce

Screenshots

Additional context

[fatcat-z]

Could you please help me understand questions below firstly? Did you try to run the converted onnx model in python? If so, does it work or not? You mentioned that 'test with onnx model converted by torch'. Do you mean you write the same model in both PyTorch and Tensorflow? Or you just test your C++ code works for model converted from a PyTorch model?

[zx-lhb]

Could you please help me understand questions below firstly? Did you try to run the converted onnx model in python? If so, does it work or not? You mentioned that 'test with onnx model converted by torch'. Do you mean you write the same model in both PyTorch and Tensorflow? Or you just test your C++ code works for model converted from a PyTorch model?

sorry, i did not give a clear discrpition. the onnx model met with errors was converted from tf2, and yes, I have run the converted onnx model in python and it woked fine. But when I run it in c++, it did not work. In order to position who bring the error, the c++ code or onnx, I test the onnx models conveted from pytorch,they worked fine. So I think it might be someting wrong with onnx model converted from tf2. But I don't know whether there are code differences between tf-onnx and torch-onnx, So I raise the question. Thank you for your reply, don't hesitate to ask if you have any question for the description.

[fatcat-z]

Probably the model logic is same, but the ONNX file we got by exporting its PyTorch version, won't be the same as the one we got by exporting its TensorFlow version. These 2 files might have different ops or different parameters for same ops.

If the ONNX model exported by tf2onnx could be run successfully in Python, then I think the issue happened in C++ with the same ONNX file should be raised to ONNXRuntime at this moment.