SuperPoint tensorrt test with tensor size mismatch error

[myboyhood]

Hello! I am learning swarm_loop package. I try to use src/loop_tensorrt_test.cpp to verify using superpoint trt to detect in my image. However I got the error of tensor size mismatch.

Problem generation:

1. I have deployed https://github.com/wang-xinyu/tensorrtx/ and https://github.com/xiang-wuu/SuperPointPretrainedNetwork
2. using python gen_wts.py to generate superpoint_v1.wts
3. I notice /tensorrtx/superpoint/superpoint.cpp at line 19-20 as following:

static const int INPUT_H = 120; // 208
static const int INPUT_W = 160; // 400 

3. then using the following command to generate supernet.engine.

 ./supernet -s SuperPointPretrainedNetwork/superpoint_v1.wts

4. My test code is as following:

#include "superpoint_tensorrt.h"
using namespace Swarm;

int main(int argc, char* argv[]) {
    if (argc < 2) {
        return -1;
    }

    std::cout << "Load Model from " << argv[1] << std::endl;
    
    std::string engine_path(argv[1]);

    SuperPointTensorRT sp_trt(engine_path, "", "",  160, 120, 0.012, true); //origin code
//    SuperPointTensorRT sp_trt(engine_path, "", "",  400, 208, 0.012, true);

    std::cout << "Load Model success" << std::endl << " Loading image " << argv[2] << std::endl;

    cv::Mat img = cv::imread(argv[2]);
    cv::resize(img, img, cv::Size(160, 120));// origin code
//    cv::resize(img, img, cv::Size(400, 208));// my change
    std::vector<float> local_desc;
    std::vector<cv::Point2f> kps;

    cv::Mat img_gray;
    cv::cvtColor(img, img_gray, cv::COLOR_BGR2GRAY);
    TicToc tic;
    for (unsigned int i = 0; i < 100; i ++) {
        sp_trt.inference(img_gray, kps, local_desc);
    }
    double dt = tic.toc();

    std::cout << "\nSuperpoint 100 takes" << dt << std::endl;
    
    for(auto pt : kps) {
        cv::circle(img, pt, 1, cv::Scalar(255, 0, 0), -1);
    }

    cv::resize(img, img, cv::Size(), 4, 4);
    cv::imshow("Image", img);
    cv::waitKey(-1);
}

5. I run the following command to test

rosrun superglue_feature_track loop_tensorrt_test /home/wzy/vins_ws/src/super-glue-pretrained-network-cius/models/weights/supernet_160_120.engine /home/wzy/vins_ws/src/super-glue-pretrained-network-cius/assets/room/lab1.png

6. In order to get more info, I add cout in tensorrt_generic.cpp to get the following terminal output

Load Model from /home/wzy/vins_ws/src/super-glue-pretrained-network-cius/models/weights/supernet_160_120.engine
Trying to init TRT engine of SuperPointTensorRT/home/wzy/vins_ws/src/super-glue-pretrained-network-cius/models/weights/supernet_160_120.engine
Loading TRT Engine...
Loading Complete!
TensorRT binding index 0 name data
TensorRT binding index 1 name semi
TensorRT binding index 2 name desc
MaxBatchSize1
Tensorsemi bind to 1 dim 65 15 20 0
Tensordesc bind to 2 dim 256 15 20 0
name:semi
inputDims.nbDims: 3
inputDims.d[i]: 65
inputDims.d[i]: 15
inputDims.d[i]: 20
19500:19200
name:desc
inputDims.nbDims: 3
inputDims.d[i]: 256
inputDims.d[i]: 15
inputDims.d[i]: 20
76800:76800

7. it throw out error as loop_tensorrt_test: /home/wzy/vins_ws/src/super-glue-pretrained-network-cius/src/tensorrt_generic.cpp:90: bool Swarm::TensorRTInferenceGeneric::verifyEngine(): Assertion get3DTensorVolume4(m_Engine->getBindingDimensions(tensor.bindingIndex)) == tensor.volume && "Tensor volumes dont match between cfg and engine file \n"' failed`

I find semi 19500:19200 is not match, I find 19500 = 651520 and 19200 = 641520. I do not know how to fix this error. Hope some help!!

[myboyhood]

The error is mainly that the tensor size is not matched when supernet.engine load and init. When we multiply the three dims of semi, we get 19500 = 65 * 15 * 20, I check 65 is specified by SuperPoint officially. but the supernet.engine load with tensor.volume = 19200 (64 * 15 * 20)

I think the problem is whether 65 or 64 is right when I use supernet.engine, How can I fix this error? much appreciation for any help !