[Bug]: Preprocessing problem with resize

[osivinyuk]

OpenVINO Version

2023.3.0

Operating System

Windows System

Device used for inference

CPU

Framework

ONNX

Model used

Yolo v7

Issue description

Preprocessing problem with YOLO v7 without resize.

Step-by-step reproduction

I am using C++ with ONNX Yolo v7 model. Model size 640x640 Preprocessing code:

v::preprocess::PrePostProcessor ppp(_model); ov::preprocess::InputInfo& input = ppp.input();

    input.tensor().set_spatial_dynamic_shape()
        .set_element_type(precision == PreprocessParameters::PrecisionU8 ? ov::element::u8 : ov::element::f32)
        //.set_color_format(ov::preprocess::ColorFormat::BGR)
        .set_layout("NHWC");
    input.preprocess().convert_element_type(ov::element::f32)
              .resize(ov::preprocess::ResizeAlgorithm::RESIZE_CUBIC);
    if(    _parameters.preprocess.shape_type == PreprocessParameters::ShapeNCHW 
        || (_parameters.preprocess.input_type == PreprocessParameters::Automatic && _parameters.preprocess.shape_type == PreprocessParameters::ShapeUnknown) )
        input.model().set_layout("NCHW");
    else if(_parameters.preprocess.shape_type == PreprocessParameters::ShapeNHWC )
    {
        input.model().set_layout("NHWC");
    }
    else 
    {
        input.model().set_layout("NCHW");
    }
    if (_parameters.preprocess.swap_rgb)
    {
        input.preprocess().reverse_channels();
    }
    if (isMeanValid(_parameters.preprocess.means, _parameters.channel_count))
    {
        input.preprocess().mean(_parameters.preprocess.means);
    }
    if ( isDividersValid(_parameters.preprocess.dividers, _parameters.channel_count) )
    {
        if (isScaleValid(_parameters.preprocess.scale))
        {
            std::vector<float> dividers(_parameters.channel_count);
            for (int channel = 0; channel < _parameters.channel_count; ++channel)
            {
                dividers[channel] = 1.0 / (_parameters.preprocess.scale / _parameters.preprocess.dividers[channel]);
            }
            input.preprocess().scale(dividers);
        }
        else
        {
            input.preprocess().scale(_parameters.preprocess.dividers);
        }
    }
    else if (isScaleValid(_parameters.preprocess.scale))
    {
        input.preprocess().scale( 1.0 / _parameters.preprocess.scale);
    }
    std::cout << ppp;
    _model = ppp.build();

If i am sending image with resolution 640x640(as model size) i am recieving fatal error(access violation).

Now I am checking image resoluton and adding 1 pixel and its working)))

Relevant log output

No response

Issue submission checklist

• [X] I'm reporting an issue. It's not a question.
• [X] I checked the problem with the documentation, FAQ, open issues, Stack Overflow, etc., and have not found a solution.
• [X] There is reproducer code and related data files such as images, videos, models, etc.

[yuxu42]

@tiger100256-hu Could you please take a look? Thanks!

[tiger100256-hu]

@osivinyuk because I don't know the details of _parameters, so could you help to provide a sample code which can reproduce this issue. below is a example code

#include <iostream>
#include "openvino/openvino.hpp"
int main() {
    ov::Core core;
    auto _model = core.read_model("model/yolov7-tiny.xml");
    ov::preprocess::PrePostProcessor ppp(_model);
    ov::preprocess::InputInfo& inputa = ppp.input();
    inputa.model().set_layout("NCHW");
    inputa.preprocess().reverse_channels();
    inputa.preprocess().mean(116.78f);
    inputa.preprocess().scale({57.21f, 57.45f, 57.73f});
    _model = ppp.build();
    auto compile_model = core.compile_model(_model, "CPU");
    auto infer_request = compile_model.create_infer_request();
    infer_request.infer();
    std::cout << "finished" << std::endl;
    return 0;
}

[osivinyuk]

could you check this: input.tensor().set_spatial_dynamic_shape() .set_element_type(precision == PreprocessParameters::PrecisionU8 ? ov::element::u8 : ov::element::f32) .set_layout("NHWC"); input.preprocess().convert_element_type(ov::element::f32) .resize(ov::preprocess::ResizeAlgorithm::RESIZE_CUBIC);

the main things are: set_spatial_dynamic_shape and resize(ov::preprocess::ResizeAlgorithm::RESIZE_CUBIC)

As i understand problem with preprocess resize

[tiger100256-hu]

@osivinyuk I try it on windows machine, below is my test code

#include <iostream>
#include "openvino/openvino.hpp"
int main() {
    ov::Core core;
    auto _model = core.read_model("model/yolov7-tiny.xml");
    ov::preprocess::PrePostProcessor ppp(_model);
    ov::preprocess::InputInfo& inputa = ppp.input();
    inputa.tensor().set_spatial_dynamic_shape();
    inputa.tensor().set_element_type(ov::element::f32);
    inputa.tensor().set_layout("NHWC");
    inputa.model().set_layout("NCHW");
    inputa.preprocess().convert_element_type(ov::element::f32);
    inputa.preprocess().reverse_channels();
    inputa.preprocess().resize(ov::preprocess::ResizeAlgorithm::RESIZE_CUBIC);
    inputa.preprocess().mean(116.78f);
    inputa.preprocess().scale({57.21f, 57.45f, 57.73f});
    _model = ppp.build();
    auto compile_model = core.compile_model(_model, "CPU");
    auto infer_request = compile_model.create_infer_request();
    infer_request.infer();
    std::cout << "finished" << std::endl;
    return 0;
}

when run with openvino 2023.3.0 release package, the code work well. but when run with openvino 2023.3.0 debug package, there is a crash, this issue is cause by line https://github.com/openvinotoolkit/openvino/blob/releases/2023/3/src/common/snippets/src/lowered/loop_manager.cpp#L367

is this the issue you met？if not, please help to provide sample code which can reproduce the issue, thanks

[osivinyuk]

its working some time and in 1-2 minutes its crash(memory heap) we see warnings about memory heap and after its crash we removed preprocessing 'set_spatial_dynamic_shape' and 'resize' and now working good

we saw two problems:

• network size input image
• some time working and crash, but with some specific image (always on the same frame)

[tiger100256-hu]

@osivinyuk thanks very much for reply, does the workflow of your application is like this

     for( int i = 0; i < image_numbers; i++) {
           open image file;
           resize image to 1*3*640*640(do by ov::preprocess::PrePostProcessor)
           do inference
     }

Does all the input images are 640*640 it's better that if your can provide sample code

[osivinyuk]

We had this issue with different resolutions. Not only 640x640. Other model was 128x128 and in some short time we had fatal error.

[yuxu42]

@osivinyuk it's better if you can provide the code for the issue replication as mentioned by @tiger100256-hu. It would be helpful to accelerate the problem solving.

[osivinyuk]

its difficult, because we are remove this part of code, but i wrote it in first message. we removed this:

input.tensor().set_spatial_dynamic_shape() .set_element_type(precision == PreprocessParameters::PrecisionU8 ? ov::element::u8 : ov::element::f32) .set_layout("NHWC"); input.preprocess().convert_element_type(ov::element::f32) .resize(ov::preprocess::ResizeAlgorithm::RESIZE_CUBIC);

and its start to work, but with our resizer

[tiger100256-hu]

@osivinyuk I am glad that your application is work now. About the resize issue in preprocessing, I will try some cases, but I am not sure that I can reproduce it.

[tiger100256-hu]

@osivinyuk sorry, I can't reproduce the resize issue. @yuxu42 about the windows debug package issue , it seemed already fixed by commit https://github.com/openvinotoolkit/openvino/commits/987cec8b173bb6169f515ff1bf9b75368e5ff311

[yuxu42]

@osivinyuk Feel free to let us know if you have further questions. Thanks!