[Bug]Can't Convert Craft to ONNX and OpenVINO IR

[shliar]

System information (version)

• OpenVINO => 2022.1
• Operating System / Platform => Windows 64 Bit
• Compiler => Visual Studio 2017
• Problem classification: Model Conversion
• Framework: Pytorch
• Model name: VGG16

Detailed description

I want to convert the craft model based on pytorch to IR model, but the tutorial is based on fastseg，but the craft model is based on vgg16. How should I convert it? you can find the craft in https://github.com/clovaai/CRAFT-pytorch.git

Steps to reproduce

I failed to convert pytorch model to onnx model, because in the example, torch onnx. The model parameter required by the export function is mobilev3large from_ pretrained(). cpu(). eval() and vgg16 doesn't seem to have a from_ pretrained(). cpu(). eval() method

[tomdol]

I've modified the bottom section of the craft.py file the following way and executed it. It has successfully created a craft.onnx file in my home directory.

if __name__ == '__main__':
    model = CRAFT(pretrained=True)
    onnx_model_path = "/home/tomdol/craft.onnx"

    model.eval()

    model_input = torch.randn(1, 3, 224, 224)

    torch.onnx.export(model,
        model_input,
        onnx_model_path,
        export_params=True,
        opset_version=11,
        do_constant_folding=False) 

[zulkifli-halim]

Hi @shliar, I am able to convert the PyTorch model to ONNX by following the step from @tomdol and also able to convert the model to IR using this command: mo --input_model craft.onnx

OpenVINO runtime version:       2022.1.0-7019-cdb9bec7210-releases/2022/1
Model Optimizer version:        2022.1.0-7019-cdb9bec7210-releases/2022/1
[ SUCCESS ] Generated IR version 11 model.
[ SUCCESS ] XML file: C:\Users\GitHub\12021\CRAFT-pytorch-master\craft.xml
[ SUCCESS ] BIN file: C:\Users\GitHub\12021\CRAFT-pytorch-master\craft.bin
[ SUCCESS ] Total execution time: 2.18 seconds.

[shliar]

我已经按照以下方式修改了文件的底部craft.py并执行了它。它已成功craft.onnx在我的主目录中创建了一个文件。

if __name__ == '__main__':
    model = CRAFT(pretrained=True)
    onnx_model_path = "/home/tomdol/craft.onnx"

    model.eval()

    model_input = torch.randn(1, 3, 224, 224)

    torch.onnx.export(model,
        model_input,
        onnx_model_path,
        export_params=True,
        opset_version=11,
        do_constant_folding=False) 

我已经按照以下方式修改了文件的底部craft.py并执行了它。它已成功craft.onnx在我的主目录中创建了一个文件。

if __name__ == '__main__':
    model = CRAFT(pretrained=True)
    onnx_model_path = "/home/tomdol/craft.onnx"

    model.eval()

    model_input = torch.randn(1, 3, 224, 224)

    torch.onnx.export(model,
        model_input,
        onnx_model_path,
        export_params=True,
        opset_version=11,
        do_constant_folding=False) 

Thank you! I have successfully transformed the onnx model

[shliar]

Hi @shliar, I am able to convert the PyTorch model to ONNX by following the step from @tomdol and also able to convert the model to IR using this command: mo --input_model craft.onnx

OpenVINO runtime version:       2022.1.0-7019-cdb9bec7210-releases/2022/1
Model Optimizer version:        2022.1.0-7019-cdb9bec7210-releases/2022/1
[ SUCCESS ] Generated IR version 11 model.
[ SUCCESS ] XML file: C:\Users\GitHub\12021\CRAFT-pytorch-master\craft.xml
[ SUCCESS ] BIN file: C:\Users\GitHub\12021\CRAFT-pytorch-master\craft.bin
[ SUCCESS ] Total execution time: 2.18 seconds.

Hi，I have successfully converted the IR model, which contains XML files and bin files, but I have encountered new difficulties. How can I perform reasoning on the IR model? In the example, segmentation_map_to_image method is used,It has only two inputs, and in the craft built with pytorch, I need net, image and text respectively_ threshold, link_ threshold, low_ text, cuda, poly, refine_ Net parameter to complete text detection,How can I use openvino to achieve the same reasoning？

[shliar]

When I try to infer the successful IR model, the following errors appear. How should I solve this problem? you can find the IR model in https://gitee.com/shliar/craft-ir-model.git

[zulkifli-halim]

@shliar. I wasn't able to access this page: https://gitee.com/shliar/craft-ir-model.git

Have you tried running the IR model with benchmark app? Does it work?

[shliar]

@shliar. I wasn't able to access this page: https://gitee.com/shliar/craft-ir-model.git

Have you tried running the IR model with benchmark app? Does it work?

Hi: I'm sorry, it seems that this is because the open source audit of the warehouse has not passed,I have put the model in the following link craft.xml: https://drive.google.com/file/d/1cjRjoawHqRL4WSBMwJZ355vMIIz1jUob/view?usp=sharing craft.bin: https://drive.google.com/file/d/12vomS1XVAxBFFaLlNTilNjkwhQoEr_hy/view?usp=sharing And I tried to use benchmark app, it works well

[zulkifli-halim]

@shliar, Can I know, which code did you use to run inference? Is it from one of the OMZ demos or a custom code? If possible, please share it with us for replication purposes.

[shliar]

@shliar, Can I know, which code did you use to run inference? Is it from one of the OMZ demos or a custom code? If possible, please share it with us for replication purposes.

Hi: I am using the following code to run inference: ie = Core() ir_path="CRAFT-pytorch-master/ir/craft.xml" model_ir = ie.read_model(model=ir_path) compiled_model_ir = ie.compile_model(model=model_ir, device_name="CPU")

Get input and output layers

output_layer_ir = compiled_model_ir.output(0)

Run inference on the input image

res_ir = compiled_model_ir([input_image])[output_layer_ir] result_mask_ir = np.squeeze(np.argmax(res_ir, axis=1)).astype(np.uint8) viz_result_image( image, segmentation_map_to_image(result=result_mask_ir, colormap=CityScapesSegmentation.get_colormap()), resize=True, ) and the result is

I know it may be incorrect to use 'CityScapesSegmentation.get_colormap()' function on the customized model, but I don't know what method to use to output colormap

[shliar]

Hi: I tried the following code: import cv2 import matplotlib.pyplot as plt import numpy as np from openvino.runtime import Core

ie = Core()

#model = ie.read_model(model="model/horizontal-text-detection-0001.xml") model = ie.read_model(model="model/craft.xml") compiled_model = ie.compile_model(model=model, device_name="CPU")

input_layer_ir = compiled_model.input(0) output_layer_ir = compiled_model.output("281") print(output_layer_ir) <ConstOutput: names[281] shape{1,32,112,112} type: f32>

Text detection models expects image in BGR format

image = cv2.imread("data/intel_rnb.jpg")

N,C,H,W = batch size, number of channels, height, width

N, C, H, W = input_layer_ir.shape

Resize image to meet network expected input sizes

resized_image = cv2.resize(image, (W, H))

Reshape to network input shape

input_image = np.expand_dims(resized_image.transpose(2, 0, 1), 0) plt.imshow(cv2.cvtColor(image, cv2.COLOR_BGR2RGB))

Create inference request

boxes = compiled_model([input_image])[output_layer_ir]

It makes this error:

IndexError Traceback (most recent call last) in 3 4 # Remove zero only boxes ----> 5 boxes = boxes[~np.all(boxes == 0, axis=1)]

IndexError: boolean index did not match indexed array along dimension 1; dimension is 32 but corresponding boolean dimension is 112

This seems to be the dimension problem of the array, but how do I solve this problem?

[zulkifli-halim]

@shliar. The error is related to the blob shape that is not matching with the model's input shape. Try reshaping the blob (a.k.a image) to fit the model's input shape, or convert the model (specifying input 706,706 instead of 224,224) to fit the blob's shape.

[shliar]

@shliar. The error is related to the blob shape that is not matching with the model's input shape. Try reshaping the blob (a.k.a image) to fit the model's input shape, or convert the model (specifying input 706,706 instead of 224,224) to fit the blob's shape.

Hi: I tried input 736,1280 which is the same as pytorch model and these are results of ir models and pytorch model after forward pass pytorch: openvino: It seems that their output is different

[shliar]

@shliar. The error is related to the blob shape that is not matching with the model's input shape. Try reshaping the blob (a.k.a image) to fit the model's input shape, or convert the model (specifying input 706,706 instead of 224,224) to fit the blob's shape.

Hi: You can find my code in https://github.com/shliar/craft-openvino-pytorch

[zulkifli-halim]

@shliar, I ran test_pytorch.py from your repo and didn't recieved the same result as yours.

Loading weights from checkpoint (C:\Users\GitHub\12021\craft-openvino-pytorch-master\models\craft_ic15_20k.pth)
elapsed time : 1657683193.5898242s

I received error when running test_openvino.py.

File "test_openvino.py", line 154, in <module>
    net.load_state_dict(copyStateDict(torch.load(args.trained_model, map_location='cpu')))
  File "C:\Users\GitHub\openvino_env\lib\site-packages\torch\serialization.py", line 595, in load
    return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
  File "C:\Users\GitHub\openvino_env\lib\site-packages\torch\serialization.py", line 764, in _legacy_load
    magic_number = pickle_module.load(f, **pickle_load_args)
_pickle.UnpicklingError: invalid load key, '<'.

You can try this command to convert craft.onnx to IR and see if it makes any changes: mo --input_model craft.onnx --input_shape [1,3,706,706] --reverse_input_channels --mean_values=[123.675,116.28,103.53] --scale_values=[58.395,57.12,57.375]

[shliar]

mo --input_model craft.onnx --input_shape [1,3,706,706] --reverse_input_channels --mean_values=[123.675,116.28,103.53] --scale_values=[58.395,57.12,57.375]

Hi: Your error can be solved by using the 'test_openvino.zip' I provided .You need to unzip it and replace the 'test_openvino.py' in https://github.com/shliar/craft-openvino-pytorch with 'test_openvino.py' in the compressed package

test_openvino.zip

And I've tried this command to convert craft.onnx to IR，but it can't makes any changes.It works, but it doesn't get any results.

In pytorch ,I can use to get two arrays after the net's forward pass,but when I try to do the same thing with IR model,the values of these two arrays look completely different this is the values in pytorch this is the values in openvino

[zulkifli-halim]

@shliar. We'll investigate further to provide some more alternatives for you and we'll get back asap.

[avitial]

@shliar apologies for the delay in our response. I've spent some time comparing the results from the torch model, onnx model (with ONNXRT) and OpenVINO (generated models based on shared https://github.com/shliar/craft-openvino-pytorch). I don't see any differences of output between ONNXRT and OpenVINO (see outputs below) while feeding the same input data (res_img_1.jpg), which leads me to conclude the issue you observe is not caused by OpenVINO.

I'd suggest to check if the exported onnx model from torch itself is perhaps getting you the same results as in the PyTorch framework. For example, before doing any post-processing simply compare the raw output data given by the respective framework. After confirming the exported ONNX model is giving the right output, then try to optimize the model with OpenVINO. If the issue persists with OpenVINO optimized model afterwards, kindly let us know and we will gladly take a look. Hope this helps!

Model Optimizer command used for your reference: $ mo --input_model craft.onnx --input_shape [1,3,736,1280] --output 291 --input input.1 --use_new_frontend

onnxrt-test

openvino-test

[avitial]

Closing this, I hope previous responses were sufficient to help you proceed. Feel free to reopen and ask additional questions related to this topic.

[rmast]

With the repo provided by @shliar I'm not able to reproduce the results of the above openvino-test. I've tried the instruction in https://github.com/shliar/craft-openvino-pytorch. There are no new commits and the test-pytorch-script contains contains strange exit-statements. I also couldn't find any other example on the internet running torchvision model vgg16_bn in Python on Intel GPU, except for this project: https://github.com/pytorch/ort with a slightly other approach. I'll try it as soon as I've time again.

ort enforces CUDA, despite the promise to support OpenVino for inference. Unfortunately Intel has ceased support for ort in January: https://github.com/pytorch/ort/commit/eb3fbb9664d7ab4d3a976cce6ad748e2d52c3705

[rmast]

If you're interested in my experiences with exporting to ONNX and seeing it run on OpenVino on a Core i5-8500: https://github.com/JaidedAI/EasyOCR/issues/746