Training for OpenCV TextDetectionModel_DB

[atagulmert]

Hi, I'm trying to train a new custom DB model and possibly convert it to ONNX format to use it with OpenCV. I'm already using pre-trained weights which are given here. I've also managed to train and get good results with eval.py and demo.py. Then I tried to convert that model file to ONNX using torch.onnx.export. Here is how I did it after loading the model and weights;

dummy_input = torch.randn(1, 3, None, None, requires_grad=True)
torch.onnx.export(model, dummy_input, "db.onnx")  

This results in an error
RuntimeError: hasSpecialCase INTERNAL ASSERT FAILED at /pytorch/torch/csrc/jit/passes/alias_analysis.cpp:300, please report a bug to PyTorch. We don't have an op for aten::to but it isn't a special case. (analyzeImpl at /pytorch/torch/csrc/jit/passes/alias_analysis.cpp:300)

To fix this in model.py(line 39) I have commented out paralleling of model.

self.model = BasicModel(args)
        # for loading models
        # self.model = parallelize(self.model, distributed, local_rank)
        self.criterion = SegDetectorLossBuilder(...

After that it exports an onnx file without an error. When I try to import that model to opencv and do inference, I get these error;

opencv/modules/core/src/matrix.cpp:806: error: (-215:Assertion failed) m.dims <= 2 in function 'Mat'

I'm not sure if the problem is occuring when converting to onnx. Problem might be opencv related. Are there any guidelines to train/use our own DB models with opencv?

[tranducanhbk]

Hi Mert Atagül!

Coulde you show CODE for convert pytorch to ONNX. Many thanks you

[atagulmert]

I used pytorch's function torch.onnx.export which you can find details here. To be more specific I have modified the demo.py to create the model and load my weights into it. Added a few lines here

def inference(self, image_path, visualize=False):
        self.init_torch_tensor()
        model = self.init_model()
        self.resume(model, self.model_path)
        dummy_input = torch.randn(1, 3, 736, 992, requires_grad=True)
        all_matircs = {}
        model.eval()
        torch.onnx.export(model, dummy_input, "db.onnx")
        batch = dict()
        batch['filename'] = [image_path]
        img, original_shape = self.load_image(image_path)
        batch['shape'] = [original_shape]
        with torch.no_grad():
            batch['image'] = img
            print(img.shape)
            pred = model.forward(batch, training=False)
            output = self.structure.representer.represent(batch, pred, is_output_polygon=self.args['polygon']) 
            if not os.path.isdir(self.args['result_dir']):
                os.mkdir(self.args['result_dir'])
            self.format_output(batch, output)

            if visualize and self.structure.visualizer:
                vis_image = self.structure.visualizer.demo_visualize(image_path, output)
                cv2.imwrite(os.path.join(self.args['result_dir'], image_path.split('/')[-1].split('.')[0]+'.jpg'), vis_image)

[tranducanhbk]

Thank you your quick reply. Maybe we have to train with non paralelnize mode. I trained with parallel mode and then load with non parallel to inference the results is bad

[atagulmert]

I doubt that data paralleling effects inference results. It would be great if someone who achieved training and transporting that model to openCV could help.

[tranducanhbk]

When i try using your code to convert torch to onnx i got error like below: But when i using https://github.com/Media-Smart/volksdep it is ok. Do you know why?

[yiran-THU]

@MhLiao I also encounter this problem, can you help us?

[yiran-THU]

@atagulmert @tranducanhbk

[yiran-THU]

Have you solve this problem?

[atagulmert]

Unfortunately no I didn't and currently working on something else. But I'm still keen to learn if this is possible.

[yiran-THU]

@atagulmert

Have change DCN module? When I follow your method, get following error: RuntimeError: ONNX export failed: Couldn't export Python operator ModulatedDeformConvFunction