openvino
openvino copied to clipboard
openvinotoolkit
[OP CONFORMANCE][TEMPLATE] Convolution tests are failed in opset conformance
Closes https://github.com/openvinotoolkit/openvino/issues/23554
Details:
- item1
- ...
Tickets:
- ticket-id
@awayzjj Many thanks for investigation and contribution to OpenVINO! Welcome to our project!
As I can see, we have two similar PR for two contributors: PR from @Vladislav-Denisov and PR from @awayzjj. So the main difference is fix static input static shape generation for different operations: first PR is for pooling ops, second one is for convolution.
The possible option is to merge 2 solution to one and develop static input generation for pooling and convolution over the solution. Second option is to merge first green PR with applied review comments as a source solution for input static shape generation, second PR should be based on first merged.
@Vladislav-Denisov @awayzjj Please discuss in PR comments?
FYI: Each of @Vladislav-Denisov @awayzjj will be a contributor to OpenVino. We need to decide merge order and align the solutions.
@iefode Thank you for your comments! I have incorporated your suggestions. Can you please review this PR again?
@awayzjj @Vladislav-Denisov Please check the comments under each of your PRs.
Many thanks!
@iefode sorry for too many extra includes, I have incorporated your suggestions. Can you please review this PR again?
@iefode Hi, the Conformance(OP) job failed, but failed tests did not include the failed tests described in this issue, is it ok ?
@iefode Hi, the Conformance(OP) job failed, but failed tests did not include the failed tests described in this issue, is it ok ?
We implement this functionality for conformance. This statement means that we need to fix all conformance jobs :)
@iefode Hi, the Conformance(OP) job failed, but failed tests did not include the failed tests described in this issue, is it ok ?
We implement this functionality for conformance. This statement means that we need to fix all conformance jobs :)
@iefode
Thank you for your response! I'm a bit confused. The failed tests only seem to cover the CPU plugin, as shown here:
However, the issues seem to require addressing tests of the TEMPLATE plugin, as shown here:
Why did the Conformance(OP) job not cover the TEMPLATE plugin, and do I need to fix these tests of the CPU plugin?
@iefode Hi, the Conformance(OP) job failed, but failed tests did not include the failed tests described in this issue, is it ok ?
We implement this functionality for conformance. This statement means that we need to fix all conformance jobs :)
@iefode Thank you for your response! I'm a bit confused. The failed tests only seem to cover the CPU plugin, as shown here:
However, the issues seem to require addressing tests of the TEMPLATE plugin, as shown here:
Why did the Conformance(OP) job not cover the TEMPLATE plugin, and do I need to fix these tests of the CPU plugin?
@awayzjj
Conformance tests are the same for all plugins (Including Template and CPU plugin), because they run the same executable file with the same conformance IRs over different devices. So in case fix of conformance/test infrastructure (like in our case), it should work for all plugins (the fix should improve the results, not corrupt). Have you checked the failures? Thanks!
@iefode I almost addressing all failed tests, which can be categorizing into four types:
- ConvolutionBackpropData Operation
- Convolution Operation with kernel dims>3
- Kernel of Convolution Operation with attribute setting
convert_to_const="false"in .meta file. - Cases corrupted by unproper optimization changes
if (range > std::numeric_limits<char>::max()) {
ov::test::utils::fill_data_random(&range, 1, std::numeric_limits<char>::max(), s.get_min_length(), 1);
}
ov::test::utils::fill_data_random(&dimValue, 1, range, s.get_min_length(), 1);
->
if (range > std::numeric_limits<char>::max()) {
ov::test::utils::fill_data_random(&range, 1, std::numeric_limits<char>::max(), s.get_min_length(), 1);
} else {
ov::test::utils::fill_data_random(&dimValue, 1, range, s.get_min_length(), 1);
}
Howerver, I failed to fix the test case below
openvino/bin/intel64/Release/ov_op_conformance_tests --device=CPU --input_folders=openvino/src/tests/test_utils/functional_test_utils/layer_tests_summary/temp/models/conformance_ir_files.lst, --report_unique_name --output_folder="openvino/src/tests/test_utils/functional_test_utils/layer_tests_summary/temp/report/parallel" --gtest_filter="conformance_Convolution/ReadIRTest.Inference/Op=Convolution.1_Type=f32_Shape=dynamic_IR=27d1a1cfdbadd9a8c2d0269f6177d6aabd55320aafe9a0047e90681dcad1cbe9_Device=CPU_Config=()"
It resulted in the following error:
MEM_USAGE=44504KB
[ CONFORMANCE ] Influence coefficient: 5.82331e-06
[ PLUGIN ] `SubgraphBaseTest::compile_model()` is started
[ PLUGIN ] `SubgraphBaseTest::compile_model()` is finished successfully. Duration is 0.0098428s
[ PLUGIN ] `SubgraphBaseTest::get_plugin_outputs()` is started
[ REFERENCE ] `SubgraphBaseTest::calculate_refs()` is started
[ REFERENCE ] Calculate reference in runtime
[ REFERENCE ] `SubgraphBaseTest::calculate_refs()` is started
[ PLUGIN ] `SubgraphBaseTest::get_plugin_outputs()` is finished successfully. Duration is 0.0022608s
[ REFERENCE ] `SubgraphBaseTest::calculate_refs()` is finished successfully. Duration is 0.0168911s
[ REFERENCE ] `SubgraphBaseTest::calculate_refs()` is finished successfully. Duration is 0.0170173s
[ COMPARATION ] `ov_tensor_utils.hpp::compare()` is started
[ COMPARATION ] `ov_tensor_utils.hpp::compare()` is finished successfully. Duration is 0.00046426s
[ PLUGIN ] `SubgraphBaseTest::get_plugin_outputs()` is started
[ REFERENCE ] `SubgraphBaseTest::calculate_refs()` is started
[ REFERENCE ] Calculate reference in runtime
[ REFERENCE ] `SubgraphBaseTest::calculate_refs()` is started
[ PLUGIN ] `SubgraphBaseTest::get_plugin_outputs()` is finished successfully. Duration is 0.470841s
[ REFERENCE ] `SubgraphBaseTest::calculate_refs()` is finished successfully. Duration is 24.4001s
[ REFERENCE ] `SubgraphBaseTest::calculate_refs()` is finished successfully. Duration is 24.4002s
[ COMPARATION ] `ov_tensor_utils.hpp::compare()` is started
[ COMPARATION ] `ov_tensor_utils.hpp::compare()` is finished successfully. Duration is 0.0936297s
[ PLUGIN ] `SubgraphBaseTest::get_plugin_outputs()` is started
[ REFERENCE ] `SubgraphBaseTest::calculate_refs()` is started
[ REFERENCE ] Calculate reference in runtime
[ REFERENCE ] `SubgraphBaseTest::calculate_refs()` is started
Unexpected application crash with code: 15
Upon further investigation, I found that the issue might be related to the input shape being too large for my 16GB machine.(The test case exit when the RES of the process reached about 10.8g)
The maximum shape of the input is (127, 512, 127, 127). When I restricted the shape to (64, 512, 127, 127), the test case passed.
Could you please review the changes and approved the CI pipeline run again? Thank you very much!
@iefode Hi, could you please review the changes and approved the CI pipeline run again? Thank you very much!
@iefode Hi, could you please review the changes and approved the CI pipeline run again? Thank you very much!
@awayzjj Many thanks for an exploration! So sorry for late reply. I have checked the changes and run CI pipelines
@iefode The Conformance(OP) job passed! I fixed the align issue. Could you please run the CI pipelines include ci/jenkins again?
@iefode @andrei-kochin Hi, the full scope of CI is full green, could you please merge the PR into master branch? 😀