new cmake for test+examples causes mass failures and unusable test run times

[diablodale]

Running a full test+example suite on develop now causes widespread failures and takes ~one hour to run. That length of time and number of failures makes it unusable for me to reasonably test.

Setup

• Microsoft Windows [Version 10.0.19044.1526]
• VS2019 v16.11.10
• depthai-core develop at 3a2961de709faf44313d7af4f62314ad063acdec
• OAK-D

Repro

1. cmake config, build with: x64, shared lib, debug, all examples and tests
2. ctest

Result

About an hour later (too long) massive failures across tests.

...
[ctest] 164/164 Test #164: apriltag_rgb_poe ..............................***Failed   16.12 sec
[ctest] -- found -P
[ctest] -- arguments: C:/njs/depthai-core/build/examples/apriltag_rgb.exe;
[ctest] Stack trace (most recent call last):
[ctest] #9    Object "", at 00007FFBC87B7034, in BaseThreadInitThunk
[ctest] #8    Source "d:\a01\_work\12\s\src\vctools\crt\vcstartup\src\startup\exe_main.cpp", line 17, in mainCRTStartup [00007FF77AA19FAE]
[ctest] #7    Source "d:\a01\_work\12\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl", line 331, in __scrt_common_main [00007FF77AA19C7E]
[ctest] #6    Source "d:\a01\_work\12\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl", line 288, in __scrt_common_main_seh [00007FF77AA19DBE]
[ctest] #5    Source "d:\a01\_work\12\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl", line 79, in invoke_main [00007FF77AA19F19]
[ctest] #4    Source "C:\njs\depthai-core\examples\AprilTag\apriltag_rgb.cpp", line 44, in main [00007FF77A1A723B]
[ctest]          41:     aprilTag->inputImage.setQueueSize(1);
[ctest]          42: 
[ctest]          43:     // Connect to device and start pipeline
[ctest]       >  44:     dai::Device device(pipeline);
[ctest]          45: 
[ctest]          46:     // Output queue will be used to get the mono frames from the outputs defined above
[ctest]          47:     auto manipQueue = device.getOutputQueue("aprilTagImage", 8, false);
[ctest] #3    Source "C:\njs\depthai-core\src\device\Device.cpp", line 26, in dai::Device::Device [00007FFB5766BBA5]
[ctest]          23: // Common explicit instantiation, to remove the need to define in header
[ctest]          24: constexpr std::size_t Device::EVENT_QUEUE_MAXIMUM_SIZE;
[ctest]          25: 
[ctest]       >  26: Device::Device(const Pipeline& pipeline) : DeviceBase(pipeline.getOpenVINOVersion()) {
[ctest]          27:     tryStartPipeline(pipeline);
[ctest]          28: }
[ctest] #2    Source "C:\njs\depthai-core\src\device\DeviceBase.cpp", line 287, in dai::DeviceBase::DeviceBase [00007FFB57688140]
[ctest]         285:     // If no device found, throw
[ctest]         286:     if(!found) throw std::runtime_error("No available devices");
[ctest]       > 287:     init(version, false, "");
[ctest]         288: }
[ctest]         289: 
[ctest]         290: DeviceBase::DeviceBase(OpenVINO::Version version, const char* pathToCmd) {
[ctest] #1    Object "", at 00007FFBA965B460, in CxxThrowException
[ctest] #0    Object "", at 00007FFBC69E4F69, in RaiseException
[ctest] -- After process executed,  produced the following exit code: -529697949
[ctest] CMake Error at C:/njs/depthai-core/examples/cmake/ExecuteTestTimeout.cmake:42 (message):
[ctest]    produced an error (-529697949) while running
[ctest] 
[ctest] 
[ctest] 
[ctest] 
[ctest] 57% tests passed, 71 tests failed out of 164
[ctest] 
[ctest] Label Time Summary:
[ctest] poe    = 1653.36 sec*proc (82 tests)
[ctest] usb    = 914.43 sec*proc (82 tests)
[ctest] 
[ctest] Total Test time (real) = 2567.99 sec
[ctest] 
[ctest] The following tests FAILED:
[ctest] 	  3 - color_camera_node_test_poe (Failed)
[ctest] 	  4 - color_camera_node_test_conforming_poe (Failed)
[ctest] 	  7 - image_manip_node_test_poe (Failed)
[ctest] 	  8 - image_manip_node_test_conforming_poe (Failed)
[ctest] 	 15 - neural_network_test_poe (Failed)
[ctest] 	 16 - neural_network_test_conforming_poe (Failed)
[ctest] 	 19 - openvino_blob_test_poe (Failed)
[ctest] 	 20 - openvino_blob_test_conforming_poe (Failed)
[ctest] 	 27 - device_usbspeed_test_poe (Failed)
[ctest] 	 28 - device_usbspeed_test_conforming_poe (Failed)
[ctest] 	 31 - unlimited_io_connection_test_poe (Failed)
[ctest] 	 32 - unlimited_io_connection_test_conforming_poe (Failed)
[ctest] 	 39 - multiple_devices_test_poe (Failed)
[ctest] 	 40 - multiple_devices_test_conforming_poe (Failed)
[ctest] 	 52 - rgb_camera_control_poe (Failed)
[ctest] 	 54 - rgb_preview_poe (Failed)
[ctest] 	 56 - rgb_video_poe (Failed)
[ctest] 	 58 - edge_detector_poe (Failed)
[ctest] 	 60 - feature_detector_poe (Failed)
[ctest] 	 62 - feature_tracker_poe (Failed)
[ctest] 	 64 - device_queue_event_poe (Failed)
[ctest] 	 66 - opencv_support_poe (Failed)
[ctest] 	 68 - queue_add_callback_poe (Failed)
[ctest] 	 70 - image_manip_poe (Failed)
[ctest] 	 72 - image_manip_rotate_poe (Failed)
[ctest] 	 74 - image_manip_tiling_poe (Failed)
[ctest] 	 76 - rgb_rotate_warp_poe (Failed)
[ctest] 	 78 - imu_gyroscope_accelerometer_poe (Failed)
[ctest] 	 80 - imu_rotation_vector_poe (Failed)
[ctest] 	 82 - mono_depth_mobilenetssd_poe (Failed)
[ctest] 	 84 - rgb_encoding_mono_mobilenet_poe (Failed)
[ctest] 	 86 - rgb_encoding_mono_mobilenet_depth_poe (Failed)
[ctest] 	 88 - rgb_encoding_mobilenet_poe (Failed)
[ctest] 	 90 - camera_mobilenet_sync_poe (Failed)
[ctest] 	 92 - rgb_mobilenet_poe (Failed)
[ctest] 	 94 - rgb_mobilenet_4k_poe (Failed)
[ctest] 	 96 - mono_mobilenet_poe (Failed)
[ctest] 	 98 - video_mobilenet_poe (Failed)
[ctest] 	100 - mono_camera_control_poe (Failed)
[ctest] 	102 - mono_preview_poe (Failed)
[ctest] 	104 - mono_full_resolution_saver_poe (Failed)
[ctest] 	106 - concatenate_poe (Failed)
[ctest] 	108 - normalize_poe (Failed)
[ctest] 	110 - object_tracker_poe (Failed)
[ctest] 	112 - spatial_object_tracker_poe (Failed)
[ctest] 	114 - object_tracker_video_poe (Failed)
[ctest] 	116 - script_camera_control_poe (Failed)
[ctest] 	118 - script_forward_frames_poe (Failed)
[ctest] 	120 - script_nndata_example_poe (Failed)
[ctest] 	122 - script_json_communication_poe (Failed)
[ctest] 	124 - spatial_location_calculator_poe (Failed)
[ctest] 	126 - spatial_mobilenet_mono_poe (Failed)
[ctest] 	128 - spatial_mobilenet_poe (Failed)
[ctest] 	130 - spatial_tiny_yolo_v3_poe (Failed)
[ctest] 	132 - spatial_tiny_yolo_v4_poe (Failed)
[ctest] 	134 - depth_crop_control_poe (Failed)
[ctest] 	136 - depth_preview_poe (Failed)
[ctest] 	138 - depth_post_processing_poe (Failed)
[ctest] 	140 - rgb_depth_aligned_poe (Failed)
[ctest] 	142 - stereo_depth_video_poe (Failed)
[ctest] 	144 - system_information_poe (Failed)
[ctest] 	146 - disparity_encoding_poe (Failed)
[ctest] 	148 - rgb_encoding_poe (Failed)
[ctest] 	150 - rgb_full_resolution_saver_poe (Failed)
[ctest] 	152 - rgb_mono_encoding_poe (Failed)
[ctest] 	154 - encoding_max_limit_poe (Failed)
[ctest] 	156 - mjpeg_encoding_poe (Failed)
[ctest] 	158 - tiny_yolo_v4_device_side_decoding_poe (Failed)
[ctest] 	160 - tiny_yolo_v3_device_side_decoding_poe (Failed)
[ctest] 	162 - apriltag_poe (Failed)
[ctest] 	164 - apriltag_rgb_poe (Failed)
[ctest] Errors while running CTest
[ctest] CTest finished with return code 8

Expected

Maybe 30 minutes for a test+example suite. All tests to pass or be skipped. No failures.

Notes

This is directly related to the new approach for PoE and the two preprocessor conformance.

The full suite duplicates tests for PoE, and within tests also repeatedly tests for PoE in loops when it knew on the 1st try there is no PoE. I recommend reworking the PoE approach so that it skips all PoE tests if there is no PoE attached. This could be done with a single test start check, or with a environ define sent into the test suite.

I get that your team also wants to extensively test the serialization. I doubt it is necessary to duplicate every test for both compliance and non-comply on MSVC on every test suite run. I would suggest maybe 10% (20% max) of the tests be run in this duplication and the remaining only run when a detailed environ flag is sent into the test suite.

This idea of test switches/flags is very common in CI. For example stress, perf, full, detailed, quick, etc.

[themarpe]

@diablodale Sorry for not mentioning explicitly, will update README.md with the following information:

The tests are now grouped by labels to specify on what device certain test should run.

ctest -L usb will only run tests that can run on USB devices, ctest -L poe will only run tests that can run on PoE devices, and so forth. (More to come)

This was done to be able to not do tests for devices not at hand, and/or to limit running tests to only certain kind of devices reachable.

Regarding MSVC conforming tests, that could be left out to just a couple more specific tests, I agree. But IMO not too big of an issue at the moment as the number of tests isn't that big. Will refactor if it grows too big.

[themarpe]

This could be done with a single test start check, or with a environ define sent into the test suite.

Interesting concept, to first discover capabilities/devices and then execute test groups that can be carried out in this scenario.

Does CTest support anything in this regard, paired with labels for instance?

[diablodale]

So many conforming tests... 2x... so slow... 🥱😉 Please consider it a request to lessen the time needed in the typical (non-full/all) test configuration. There can always be a "full" option for those that want extensive coverage.

A few ideas some to mind for auto selecting test groups

1. Do it during cmake config. I believe the majority of devs will know during cmake config their hardware setup. There is already DEPTHAI_BUILD_TESTS. The interpretation can be changed (or new var) to be the group(s) of tests to run. When tests/Cmakefile.txt is processed during config, it can use DEPTHAI_BUILD_TESTS to add/exclude tests.
2. Run a dedicated EXE that will do a simple probe for device types available, write the result to stdout or tmp, and then use that as params to into ctest like -R, -E, or -L, params to test EXEs, or env vars.
3. Use CTEST_CUSTOM_PRE_TEST and/or CTestCustom.cmake as per https://gitlab.kitware.com/cmake/community/-/wikis/doc/ctest/Testing-With-CTest#customizing-ctest