[Good First Issue]: [OP CONFORMANCE][TEMPLATE] One Multiply test in failed in confomance over template

[iefode]

Context

OP conformance suite is validation tool checks a plugin conditions from operation implementation status perspective. OP conformance is based on extracted from OMZ model scope operations and graphs by ov_subgraphs_dumper tool. Extracted graphs are saved as IR (OpenVINO Intermediate representation) and is stored in the public share. OP conformance suite contains the following test types:

• OpImplCheck validates operation support by plugin. Return true or false.
• Inference compare device inference results vs reference over extracted IR as a model. We generate synthetic tensors to get a result. Possible results are passed, failed, hanged (interrupted by timeout), crashed and skipped (just in case -shape_mode is misaligned with graph inputs).
• QueryModel check possibility of graph execution per device. Have the same status as Inference
• ImportExport exports compiled model, imports it and check that models are same. Have the same status as Inference

Template is a simple plugin run inference using reference implementation. It means if we will run conformance over template, we should compare TEMPLATE vs TEMPLATE results. Sometimes we have bugs inside the plugin, reference implementation or test infrastructure leads to negative test result.

What needs to be done?

Description: conformance_Multiply/ReadIRTest.Inference/Op=Multiply.1_Type=f32_Shape=dynamic_IR=287a7562757ef0295cc38442e3d775cff0fb1ea9b27e6897bd456f01ce82d455_Device=TEMPLATE_Config=() is failed.

How to reproduce:

1. Build the OV using -DENABLE_TESTS=ON -DENABLE_FUNACTIONAL_TESTS=ON
2. Build ov_op_conformance_tests target
3. Run OP conformance to download conformance IRs and run executable file with args:

python3 /openvino/src/tests/test_utils/functional_test_utils/layer_tests_summary/run_conformance.py -d=TEMPLATE --gtest_filter="conformance_Multiply/ReadIRTest.Inference/Op=Multiply.1_Type=f32_Shape=dynamic_IR=287a7562757ef0295cc38442e3d775cff0fb1ea9b27e6897bd456f01ce82d455_Device=TEMPLATE_Config=()"

NOTE: run_conformance log contains all commands to run ov_op_conformance_tests! You can take them to debug!

4. Check the logs inside working directory and get the results:

[ RUN      ] conformance_Multiply/ReadIRTest.Inference/Op=Multiply.1_Type=f32_Shape=dynamic_IR=287a7562757ef0295cc38442e3d775cff0fb1ea9b27e6897bd456f01ce82d455_Device=TEMPLATE_Config=()

MEM_USAGE=55352KB
[ CONFORMANCE ] Influence coefficient: 2.89071e-06
[ PLUGIN      ] `SubgraphBaseTest::compile_model()` is started
[ PLUGIN      ] `SubgraphBaseTest::compile_model()` is finished successfully. Duration is 0.00832317s
[ PLUGIN      ] `SubgraphBaseTest::get_plugin_outputs()` is started
[ REFERENCE   ] `SubgraphBaseTest::calculate_refs()` is started
[ REFERENCE   ] Calculate reference in runtime
[ REFERENCE   ] `SubgraphBaseTest::calculate_refs()` is started
[ REFERENCE   ] `SubgraphBaseTest::calculate_refs()` is finished successfully. Duration is 0.00406761s
[ PLUGIN      ] `SubgraphBaseTest::get_plugin_outputs()` is finished successfully. Duration is 0.00915385s
[ REFERENCE   ] `SubgraphBaseTest::calculate_refs()` is finished successfully. Duration is 0.0157838s
[ COMPARATION ] `ov_tensor_utils.hpp::compare()` is started
[ COMPARATION ] rel_threshold: 0.639893
[ COMPARATION ] abs_threshold: 0.639893
[ COMPARATION ] `ov_tensor_utils.hpp::compare()` is finished successfully. Duration is 0.00117577s
[ REFERENCE   ] `SubgraphBaseTest::calculate_refs()` is started
[ REFERENCE   ] Calculate reference in runtime
[ REFERENCE   ] `SubgraphBaseTest::calculate_refs()` is started
[ PLUGIN      ] `SubgraphBaseTest::get_plugin_outputs()` is started
[ PLUGIN      ] `SubgraphBaseTest::get_plugin_outputs()` is finished successfully. Duration is 15.895s
[ REFERENCE   ] `SubgraphBaseTest::calculate_refs()` is finished successfully. Duration is 16.7522s
[ REFERENCE   ] `SubgraphBaseTest::calculate_refs()` is finished successfully. Duration is 16.8706s
[ COMPARATION ] `ov_tensor_utils.hpp::compare()` is started
[ COMPARATION ] rel_threshold: 0.601611
[ COMPARATION ] abs_threshold: 0.601611
[ COMPARATION ] `ov_tensor_utils.hpp::compare()` is finished successfully. Duration is 13.7659s
src/tests/functional/shared_test_classes/src/base/ov_subgraph.cpp:96: Failure
Exception from src/core/src/runtime/allocator.cpp:69:
std::bad_alloc

[  FAILED  ] conformance_Multiply/ReadIRTest.Inference/Op=Multiply.1_Type=f32_Shape=dynamic_IR=287a7562757ef0295cc38442e3d775cff0fb1ea9b27e6897bd456f01ce82d455_Device=TEMPLATE_Config=(), where GetParam() = (("/home/efode/repo/openvino/src/tests/test_utils/functional_test_utils/layer_tests_summary/temp/template_conformance/models/2023.3.0-13657-d5b0f4d2d73/operation/dynamic/Multiply-1/f32/287a7562757ef0295cc38442e3d775cff0fb1ea9b27e6897bd456f01ce82d455.xml", ""), "TEMPLATE", {}) (39759 ms)

5. To debug the test in C++ Use the following command (just an example, refer to note in third item):

openvino/bin/intel64/Release/ov_op_conformance_tests --device=TEMPLATE --input_folders=openvino/src/tests/test_utils/functional_test_utils/layer_tests_summary/temp/models/conformance_ir_files.lst, --report_unique_name --output_folder="openvino/src/tests/test_utils/functional_test_utils/layer_tests_summary/temp/report/parallel" --gtest_filter="conformance_Multiply/ReadIRTest.Inference/Op=Multiply.1_Type=f32_Shape=dynamic_IR=287a7562757ef0295cc38442e3d775cff0fb1ea9b27e6897bd456f01ce82d455_Device=TEMPLATE_Config=()" --config_path="" --shape_mode=

Expected result: Passed status for mentioned test.

Example Pull Requests

No response

Resources

• Contribution guide - start here!
• Intel DevHub Discord channel - engage in discussions, ask questions and talk to OpenVINO developers
• Conformance readme
• TEMPLATE plugin

Contact points

@iefode

Ticket

No response

[awayzjj]

.take

[github-actions[bot]]

Thank you for looking into this issue! Please let us know if you have any questions or require any help.

[YutingGao7]

.take

[github-actions[bot]]

Thank you for looking into this issue! Please let us know if you have any questions or require any help.

[p-wysocki]

Hello @YutingGao7, are you still working on this?

[iefode]

Hello @YutingGao7!

What is the status of the issue? Please, let us know in case of any issues

We will unassign the ticket from you in case of inaction

[Vladislav-Denisov]

.take

[github-actions[bot]]

Thank you for looking into this issue! Please let us know if you have any questions or require any help.

[Vladislav-Denisov]

@iefode I've done some research and I'm sharing the results with you:

When executing a loop https://github.com/openvinotoolkit/openvino/blob/6e9ee07dbc3c68fff69e72d1fe628a8e4136bc1c/src/tests/functional/shared_test_classes/src/base/ov_subgraph.cpp#L75-L78

Iterations for targetStaticShapes[0] and targetStaticShapes[1] are successful. However, when calling generate_inputs() for targetStaticShapes[2] a "bad allocation" exception occurs.

The error occurs when allocating memory for the tensor [127,512,127,127,64].

I think it is necessary to change the logic of Shape generation to solve the problem. The generate() function https://github.com/openvinotoolkit/openvino/blob/6e9ee07dbc3c68fff69e72d1fe628a8e4136bc1c/src/tests/functional/plugin/conformance/test_runner/op_conformance_runner/src/utils/generate_static_shapes.cpp#L51 must according to the principle of "taking into account the available memory capacity when generating Shape".

[mlukasze]

@iefode your feedback please

[iefode]

@mlukasze Thanks for notification!

@Vladislav-Denisov Many thanks for investigation! Looks like we should fix shape generate() method. It looks absolutely ok

[p-wysocki]

@iefode does this investigation close the ticket or should we keep it open for next steps?