Intermittent build error during the `cmake build All` step with a `Process terminated due to timeout` error when building `core/azure-core-amqp/test/ut_uamqp/Release/azure-core-amqp-uamqp-tests.exe`

[ahsonkhan]

For build leg Validate Win2019_x64_Release_Curl, see: https://dev.azure.com/azure-sdk/public/_build/results?buildId=3485567&view=logs&j=cf190778-6463-53b7-8403-1b11d749bb88&t=7254fd27-955c-5b6b-cc10-65f278d4cbcf&l=643

2024-02-09T23:18:06.3620398Z   Generating Code...
2024-02-09T23:18:17.4231086Z   CMake Error at C:/Program Files/CMake/share/cmake-3.27/Modules/GoogleTestAddTests.cmake:112 (message):
2024-02-09T23:18:17.4270245Z     Error running test executable.
2024-02-09T23:18:17.4279543Z   
2024-02-09T23:18:17.4284505Z       Path: 'D:/a/_work/1/s/build/sdk/core/azure-core-amqp/test/ut_uamqp/Release/azure-core-amqp-uamqp-tests.exe'
2024-02-09T23:18:17.4289334Z       Result: Process terminated due to timeout
2024-02-09T23:18:17.4306533Z       Output:
2024-02-09T23:18:17.4314858Z         
2024-02-09T23:18:17.4322422Z   
2024-02-09T23:18:17.4330908Z   Call Stack (most recent call first):
2024-02-09T23:18:17.4335315Z     C:/Program Files/CMake/share/cmake-3.27/Modules/GoogleTestAddTests.cmake:225 (gtest_discover_tests_impl)
2024-02-09T23:18:17.4339376Z   
2024-02-09T23:18:17.4346596Z   
2024-02-09T23:18:17.4402716Z   Building Custom Rule D:/a/_work/1/s/sdk/storage/azure-storage-common/test/ut/CMakeLists.txt
2024-02-09T23:18:17.4405095Z   Building Custom Rule D:/a/_work/1/s/sdk/storage/azure-storage-blobs/test/fault-injector/CMakeLists.txt
2024-02-09T23:18:17.5421959Z C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\MSBuild\Microsoft\VC\v160\Microsoft.CppCommon.targets(155,5): error MSB3073: The command "setlocal [D:\a\_work\1\s\build\sdk\core\azure-core-amqp\test\ut_uamqp\azure-core-amqp-uamqp-tests.vcxproj]
2024-02-09T23:18:17.5469621Z C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\MSBuild\Microsoft\VC\v160\Microsoft.CppCommon.targets(155,5): error MSB3073: "C:\Program Files\PowerShell\7\pwsh.exe" -noprofile -executionpolicy Bypass -file D:/a/_work/1/s/build/_deps/vcpkg-src/scripts/buildsystems/msbuild/applocal.ps1 -targetBinary D:/a/_work/1/s/build/sdk/core/azure-core-amqp/test/ut_uamqp/Release/azure-core-amqp-uamqp-tests.exe -installedDir D:/a/_work/1/s/build/vcpkg_installed/x64-windows-static/bin -OutVariable out [D:\a\_work\1\s\build\sdk\core\azure-core-amqp\test\ut_uamqp\azure-core-amqp-uamqp-tests.vcxproj]
...
...
2024-02-09T23:21:52.5700293Z   Building Custom Rule D:/a/_work/1/s/sdk/eventhubs/azure-messaging-eventhubs-checkpointstore-blob/test/ut/CMakeLists.txt
2024-02-09T23:21:52.6652298Z   blob_checkpoint_store_test.cpp
2024-02-09T23:21:53.1663158Z   Generating Code...
2024-02-09T23:21:58.0733307Z   azure-security-keyvault-administration-test.vcxproj -> D:\a\_work\1\s\build\sdk\keyvault\azure-security-keyvault-administration\test\ut\Release\azure-security-keyvault-administration-test.exe
2024-02-09T23:21:58.2128901Z   azure-messaging-eventhubs-blobstore-test.vcxproj -> D:\a\_work\1\s\build\sdk\eventhubs\azure-messaging-eventhubs-checkpointstore-blob\test\ut\Release\azure-messaging-eventhubs-blobstore-test.exe
2024-02-09T23:21:59.0211215Z ##[error]Cmd.exe exited with code '1'.
2024-02-09T23:21:59.0578424Z ##[section]Finishing: cmake build All

From an unrelated PR: https://github.com/Azure/azure-sdk-for-cpp/pull/5336

cc @danieljurek, @LarryOsterman

[ahsonkhan]

We have seen a significant rise in the cmake build All step failing recently. In some cases, it has required 2+ retries to get them to pass, and that is a pain-point when it occurs during releases.

This is the second biggest category of failures impacting our live test success rate, noticeably (~40%).

https://dev.azure.com/azure-sdk/internal/_pipeline/analytics/stageawareoutcome?definitionId=1615&contextType=build

During the most recent release milestone, we ended up restarting the job 4 times to get past this error: https://dev.azure.com/azure-sdk/internal/_build/results?buildId=3776048&view=logs&j=064b4c6b-f4a0-520a-fd03-31b8f122ef20&t=066de52a-2d1d-58d3-386e-7a91f4092f02&s=ce8af72c-1f05-5f4e-92a5-0b2c5d33786c

Example on Validate Win2022_Win32Api_release_curl_x86: https://dev.azure.com/azure-sdk/internal/_build/results?buildId=3783545&view=logs&j=e725447c-1a69-585a-f6f0-491e0d04b156&t=94b17bc9-80f0-58bf-d973-7a01b4c126a4

...
2024-05-13T17:35:51.0325981Z   Generating Code...
2024-05-13T17:35:55.8649100Z   blob_query_test.cpp
2024-05-13T17:35:56.6008505Z   CMake Error at C:/Program Files/CMake/share/cmake-3.29/Modules/GoogleTestAddTests.cmake:112 (message):
2024-05-13T17:35:57.1880383Z   Building Custom Rule D:/a/_work/1/s/sdk/keyvault/azure-security-keyvault-administration/test/ut/CMakeLists.txt
2024-05-13T17:36:00.5412739Z     Error running test executable.
2024-05-13T17:36:00.5445682Z   
2024-05-13T17:36:00.5645060Z       Path: 'D:/a/_work/1/s/build/sdk/core/azure-core-amqp/test/ut_uamqp/Release/azure-core-amqp-uamqp-tests.exe'
2024-05-13T17:36:03.1103286Z       Working directory: 'D:/a/_work/1/s/build/sdk/core/azure-core-amqp/test/ut_uamqp'
2024-05-13T17:36:03.1705877Z   azure-security-keyvault-certificates-test.vcxproj -> D:\a\_work\1\s\build\sdk\keyvault\azure-security-keyvault-certificates\test\ut\Release\azure-security-keyvault-certificates-test.exe
2024-05-13T17:36:03.5467689Z       Result: Process terminated due to timeout
2024-05-13T17:36:03.5497168Z       Output:
2024-05-13T17:36:03.5679522Z         
2024-05-13T17:36:03.5713353Z   
2024-05-13T17:36:03.5740072Z   blob_sas_test.cpp
2024-05-13T17:36:03.5769479Z   Call Stack (most recent call first):
2024-05-13T17:36:03.5792361Z     C:/Program Files/CMake/share/cmake-3.29/Modules/GoogleTestAddTests.cmake:226 (gtest_discover_tests_impl)
2024-05-13T17:36:03.5826123Z   
2024-05-13T17:36:03.5957528Z   
2024-05-13T17:36:04.7669768Z C:\Program Files\Microsoft Visual Studio\2022\Enterprise\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(166,5): error MSB3073: The command "setlocal [D:\a\_work\1\s\build\sdk\core\azure-core-amqp\test\ut_uamqp\azure-core-amqp-uamqp-tests.vcxproj]
2024-05-13T17:36:05.1263935Z C:\Program Files\Microsoft Visual Studio\2022\Enterprise\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(166,5): error MSB3073: "C:\Program Files\PowerShell\7\pwsh.exe" -noprofile -executionpolicy Bypass -file D:/a/_work/1/s/build/_deps/vcpkg-src/scripts/buildsystems/msbuild/applocal.ps1 -targetBinary D:/a/_work/1/s/build/sdk/core/azure-core-amqp/test/ut_uamqp/Release/azure-core-amqp-uamqp-tests.exe -installedDir D:/a/_work/1/s/build/vcpkg_installed/x86-windows-static/bin -OutVariable out [D:\a\_work\1\s\build\sdk\core\azure-core-amqp\test\ut_uamqp\azure-core-amqp-uamqp-tests.vcxproj]
2024-05-13T17:36:05.5169523Z C:\Program Files\Microsoft Visual Studio\2022\Enterprise\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(166,5): error MSB3073: if %errorlevel% neq 0 goto :cmEnd [D:\a\_work\1\s\build\sdk\core\azure-core-amqp\test\ut_uamqp\azure-core-amqp-uamqp-tests.vcxproj]
2024-05-13T17:36:05.7393502Z C:\Program Files\Microsoft Visual Studio\2022\Enterprise\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(166,5): error MSB3073: :cmEnd [D:\a\_work\1\s\build\sdk\core\azure-core-amqp\test\ut_uamqp\azure-core-amqp-uamqp-tests.vcxproj]
2024-05-13T17:36:05.9703663Z C:\Program Files\Microsoft Visual Studio\2022\Enterprise\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(166,5): error MSB3073: endlocal & call :cmErrorLevel %errorlevel% & goto :cmDone [D:\a\_work\1\s\build\sdk\core\azure-core-amqp\test\ut_uamqp\azure-core-amqp-uamqp-tests.vcxproj]
2024-05-13T17:36:06.1772186Z C:\Program Files\Microsoft Visual Studio\2022\Enterprise\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(166,5): error MSB3073: :cmErrorLevel [D:\a\_work\1\s\build\sdk\core\azure-core-amqp\test\ut_uamqp\azure-core-amqp-uamqp-tests.vcxproj]
2024-05-13T17:36:06.3328743Z C:\Program Files\Microsoft Visual Studio\2022\Enterprise\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(166,5): error MSB3073: exit /b %1 [D:\a\_work\1\s\build\sdk\core\azure-core-amqp\test\ut_uamqp\azure-core-amqp-uamqp-tests.vcxproj]
2024-05-13T17:36:06.6374458Z C:\Program Files\Microsoft Visual Studio\2022\Enterprise\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(166,5): error MSB3073: :cmDone [D:\a\_work\1\s\build\sdk\core\azure-core-amqp\test\ut_uamqp\azure-core-amqp-uamqp-tests.vcxproj]
2024-05-13T17:36:06.9550443Z C:\Program Files\Microsoft Visual Studio\2022\Enterprise\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(166,5): error MSB3073: if %errorlevel% neq 0 goto :VCEnd [D:\a\_work\1\s\build\sdk\core\azure-core-amqp\test\ut_uamqp\azure-core-amqp-uamqp-tests.vcxproj]
2024-05-13T17:36:07.2384668Z C:\Program Files\Microsoft Visual Studio\2022\Enterprise\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(166,5): error MSB3073: setlocal [D:\a\_work\1\s\build\sdk\core\azure-core-amqp\test\ut_uamqp\azure-core-amqp-uamqp-tests.vcxproj]
2024-05-13T17:36:07.5864220Z C:\Program Files\Microsoft Visual Studio\2022\Enterprise\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(166,5): error MSB3073: "C:\Program Files\CMake\bin\cmake.exe" -D TEST_TARGET=azure-core-amqp-uamqp-tests -D TEST_EXECUTABLE=D:/a/_work/1/s/build/sdk/core/azure-core-amqp/test/ut_uamqp/Release/azure-core-amqp-uamqp-tests.exe -D TEST_EXECUTOR= -D TEST_WORKING_DIR=D:/a/_work/1/s/build/sdk/core/azure-core-amqp/test/ut_uamqp -D TEST_EXTRA_ARGS= -D TEST_PROPERTIES= -D TEST_PREFIX= -D TEST_SUFFIX= -D TEST_FILTER= -D NO_PRETTY_TYPES=FALSE -D NO_PRETTY_VALUES=FALSE -D TEST_LIST=azure-core-amqp-uamqp-tests_TESTS -D CTEST_FILE=D:/a/_work/1/s/build/sdk/core/azure-core-amqp/test/ut_uamqp/azure-core-amqp-uamqp-tests[1]_tests.cmake -D TEST_DISCOVERY_TIMEOUT=5 -D TEST_XML_OUTPUT_DIR= -P "C:/Program Files/CMake/share/cmake-3.29/Modules/GoogleTestAddTests.cmake" [D:\a\_work\1\s\build\sdk\core\azure-core-amqp\test\ut_uamqp\azure-core-amqp-uamqp-tests.vcxproj]
2024-05-13T17:36:07.8519362Z C:\Program Files\Microsoft Visual Studio\2022\Enterprise\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(166,5): error MSB3073: if %errorlevel% neq 0 goto :cmEnd [D:\a\_work\1\s\build\sdk\core\azure-core-amqp\test\ut_uamqp\azure-core-amqp-uamqp-tests.vcxproj]
2024-05-13T17:36:08.1570694Z C:\Program Files\Microsoft Visual Studio\2022\Enterprise\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(166,5): error MSB3073: :cmEnd [D:\a\_work\1\s\build\sdk\core\azure-core-amqp\test\ut_uamqp\azure-core-amqp-uamqp-tests.vcxproj]
2024-05-13T17:36:08.4404411Z C:\Program Files\Microsoft Visual Studio\2022\Enterprise\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(166,5): error MSB3073: endlocal & call :cmErrorLevel %errorlevel% & goto :cmDone [D:\a\_work\1\s\build\sdk\core\azure-core-amqp\test\ut_uamqp\azure-core-amqp-uamqp-tests.vcxproj]
2024-05-13T17:36:08.7829211Z C:\Program Files\Microsoft Visual Studio\2022\Enterprise\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(166,5): error MSB3073: :cmErrorLevel [D:\a\_work\1\s\build\sdk\core\azure-core-amqp\test\ut_uamqp\azure-core-amqp-uamqp-tests.vcxproj]
2024-05-13T17:36:09.0494520Z C:\Program Files\Microsoft Visual Studio\2022\Enterprise\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(166,5): error MSB3073: exit /b %1 [D:\a\_work\1\s\build\sdk\core\azure-core-amqp\test\ut_uamqp\azure-core-amqp-uamqp-tests.vcxproj]
2024-05-13T17:36:09.2734009Z C:\Program Files\Microsoft Visual Studio\2022\Enterprise\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(166,5): error MSB3073: :cmDone [D:\a\_work\1\s\build\sdk\core\azure-core-amqp\test\ut_uamqp\azure-core-amqp-uamqp-tests.vcxproj]
2024-05-13T17:36:09.4276829Z C:\Program Files\Microsoft Visual Studio\2022\Enterprise\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(166,5): error MSB3073: if %errorlevel% neq 0 goto :VCEnd [D:\a\_work\1\s\build\sdk\core\azure-core-amqp\test\ut_uamqp\azure-core-amqp-uamqp-tests.vcxproj]
2024-05-13T17:36:09.5962461Z C:\Program Files\Microsoft Visual Studio\2022\Enterprise\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(166,5): error MSB3073: setlocal [D:\a\_work\1\s\build\sdk\core\azure-core-amqp\test\ut_uamqp\azure-core-amqp-uamqp-tests.vcxproj]
2024-05-13T17:36:09.7904813Z C:\Program Files\Microsoft Visual Studio\2022\Enterprise\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(166,5): error MSB3073: "C:\Program Files\CMake\bin\cmake.exe" -D TEST_TARGET=azure-core-amqp-uamqp-tests -D TEST_EXECUTABLE=D:/a/_work/1/s/build/sdk/core/azure-core-amqp/test/ut_uamqp/Release/azure-core-amqp-uamqp-tests.exe -D TEST_EXECUTOR= -D TEST_WORKING_DIR=D:/a/_work/1/s/build/sdk/core/azure-core-amqp/test/ut_uamqp -D TEST_EXTRA_ARGS= -D TEST_PROPERTIES= -D TEST_PREFIX=azure-core-amqp. -D TEST_SUFFIX= -D TEST_FILTER= -D NO_PRETTY_TYPES=TRUE -D NO_PRETTY_VALUES=TRUE -D TEST_LIST=azure-core-amqp-uamqp-tests_TESTS -D CTEST_FILE=D:/a/_work/1/s/build/sdk/core/azure-core-amqp/test/ut_uamqp/azure-core-amqp-uamqp-tests[2]_tests.cmake -D TEST_DISCOVERY_TIMEOUT=600 -D TEST_XML_OUTPUT_DIR= -P "C:/Program Files/CMake/share/cmake-3.29/Modules/GoogleTestAddTests.cmake" [D:\a\_work\1\s\build\sdk\core\azure-core-amqp\test\ut_uamqp\azure-core-amqp-uamqp-tests.vcxproj]
2024-05-13T17:36:09.8668613Z C:\Program Files\Microsoft Visual Studio\2022\Enterprise\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(166,5): error MSB3073: if %errorlevel% neq 0 goto :cmEnd [D:\a\_work\1\s\build\sdk\core\azure-core-amqp\test\ut_uamqp\azure-core-amqp-uamqp-tests.vcxproj]
2024-05-13T17:36:09.9730637Z C:\Program Files\Microsoft Visual Studio\2022\Enterprise\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(166,5): error MSB3073: :cmEnd [D:\a\_work\1\s\build\sdk\core\azure-core-amqp\test\ut_uamqp\azure-core-amqp-uamqp-tests.vcxproj]
2024-05-13T17:36:09.9805198Z C:\Program Files\Microsoft Visual Studio\2022\Enterprise\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(166,5): error MSB3073: endlocal & call :cmErrorLevel %errorlevel% & goto :cmDone [D:\a\_work\1\s\build\sdk\core\azure-core-amqp\test\ut_uamqp\azure-core-amqp-uamqp-tests.vcxproj]
2024-05-13T17:36:09.9852627Z C:\Program Files\Microsoft Visual Studio\2022\Enterprise\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(166,5): error MSB3073: :cmErrorLevel [D:\a\_work\1\s\build\sdk\core\azure-core-amqp\test\ut_uamqp\azure-core-amqp-uamqp-tests.vcxproj]
2024-05-13T17:36:09.9900335Z C:\Program Files\Microsoft Visual Studio\2022\Enterprise\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(166,5): error MSB3073: exit /b %1 [D:\a\_work\1\s\build\sdk\core\azure-core-amqp\test\ut_uamqp\azure-core-amqp-uamqp-tests.vcxproj]
2024-05-13T17:36:09.9916180Z C:\Program Files\Microsoft Visual Studio\2022\Enterprise\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(166,5): error MSB3073: :cmDone [D:\a\_work\1\s\build\sdk\core\azure-core-amqp\test\ut_uamqp\azure-core-amqp-uamqp-tests.vcxproj]
2024-05-13T17:36:10.0074792Z C:\Program Files\Microsoft Visual Studio\2022\Enterprise\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(166,5): error MSB3073: if %errorlevel% neq 0 goto :VCEnd [D:\a\_work\1\s\build\sdk\core\azure-core-amqp\test\ut_uamqp\azure-core-amqp-uamqp-tests.vcxproj]
2024-05-13T17:36:10.0120442Z C:\Program Files\Microsoft Visual Studio\2022\Enterprise\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(166,5): error MSB3073: :VCEnd" exited with code 1. [D:\a\_work\1\s\build\sdk\core\azure-core-amqp\test\ut_uamqp\azure-core-amqp-uamqp-tests.vcxproj]
2024-05-13T17:36:10.0191025Z   azure-core-test.vcxproj -> D:\a\_work\1\s\build\sdk\core\azure-core\test\ut\Release\azure-core-test.exe
2024-05-13T17:36:10.0199607Z   	files'	
2024-05-13T17:36:10.0326283Z   azure-storage-common-test.vcxproj -> D:\a\_work\1\s\build\sdk\storage\azure-storage-common\test\ut\Release\azure-storage-common-test.exe
...
2024-05-13T17:39:03.2482162Z   test_base.cpp
2024-05-13T17:39:08.6537899Z   Generating Code...
2024-05-13T17:39:27.4152312Z   azure-storage-files-shares-test.vcxproj -> D:\a\_work\1\s\build\sdk\storage\azure-storage-files-shares\test\ut\Release\azure-storage-files-shares-test.exe
2024-05-13T17:39:29.3775102Z   azure-security-keyvault-keys-test.vcxproj -> D:\a\_work\1\s\build\sdk\keyvault\azure-security-keyvault-keys\test\ut\Release\azure-security-keyvault-keys-test.exe
2024-05-13T17:39:34.8747027Z   azure-storage-files-datalake-test.vcxproj -> D:\a\_work\1\s\build\sdk\storage\azure-storage-files-datalake\test\ut\Release\azure-storage-files-datalake-test.exe
2024-05-13T17:39:35.5871378Z ##[error]Cmd.exe exited with code '1'.
2024-05-13T17:39:35.6144690Z ##[section]Finishing: cmake build All

[ahsonkhan]

Fixed by https://github.com/Azure/azure-sdk-for-cpp/pull/5644