dotnet
XUnitLogChecker: `runtime-libraries-coreclr outerloop` tests are unable to find the tool DLL
Error Blob
{
"ErrorMessage": "XUnitLogChecker.dll does not exist in the expected location:",
"BuildRetry": false,
"ErrorPattern": "",
"ExcludeConsoleLog": false
}
Reproduction Steps
- Queue:
runtime-libraries-coreclr outerloop - Build example: https://dev.azure.com/dnceng-public/public/_build/results?buildId=498370&view=logs&j=631ed29b-f0dd-5b15-38a0-c6c555556dea&t=b9941189-b5a8-5df3-6b3a-0964fabb3acd
- Log example: https://helixre107v0xdcypoyl9e7f.blob.core.windows.net/dotnet-runtime-refs-pull-95898-merge-d360b025a7c14161a1/ComInterfaceGenerator.Tests/1/console.66454013.log?helixlogtype=result
- Output example:
Console log: 'ComInterfaceGenerator.Tests' from job d360b025-a7c1-4161-a1a4-19f920345fd3 workitem 57966585-bda4-4f37-8ff1-0af071d4373f (windows.81.amd64.open.rt) executed on machine a0018BJ running Windows-2012ServerR2-6.3.9600-SP0
C:\h\w\A22708CE\w\A5080926\e>taskkill.exe /f /im corerun.exe
ERROR: The process "corerun.exe" not found.
C:\h\w\A22708CE\w\A5080926\e>call RunTests.cmd --runtime-path C:\h\w\A22708CE\p
========================= Begin custom configuration settings ==============================
set __TestArchitecture=x64
set __IsXUnitLogCheckerSupported=1
========================== End custom configuration settings ===============================
----- start Wed 12/13/2023 18:40:04.47 =============== To repro directly: =====================================================
pushd C:\h\w\A22708CE\w\A5080926\e\
"C:\h\w\A22708CE\p\dotnet.exe" exec --runtimeconfig ComInterfaceGenerator.Tests.runtimeconfig.json --depsfile ComInterfaceGenerator.Tests.deps.json xunit.console.dll ComInterfaceGenerator.Tests.dll -xml testResults.xml -nologo -nocolor -trait category=OuterLoop -notrait category=IgnoreForCI -notrait category=failing
popd
===========================================================================================================
C:\h\w\A22708CE\w\A5080926\e>"C:\h\w\A22708CE\p\dotnet.exe" exec --runtimeconfig ComInterfaceGenerator.Tests.runtimeconfig.json --depsfile ComInterfaceGenerator.Tests.deps.json xunit.console.dll ComInterfaceGenerator.Tests.dll -xml testResults.xml -nologo -nocolor -trait category=OuterLoop -notrait category=IgnoreForCI -notrait category=failing
Discovering: ComInterfaceGenerator.Tests (method display = ClassAndMethod, method display options = None)
Discovered: ComInterfaceGenerator.Tests (found 0 of 45 test cases)
=== TEST EXECUTION SUMMARY ===
ComInterfaceGenerator.Tests.dll Total: 0
----- end Wed 12/13/2023 18:40:06.22 ----- exit code 0 ----------------------------------------------------------
----- start =============== XUnitLogChecker Output =====================================================
XUnitLogChecker.dll does not exist in the expected location: C:\h\w\A22708CE\p\XUnitLogChecker.dll
----- end =============== XUnitLogChecker Output - exit code 1 ===============
2023-12-13T18:40:06.788Z INFO run.py run(48) main Beginning reading of test results.
2023-12-13T18:40:06.788Z INFO run.py __init__(42) read_results Searching 'C:\h\w\A22708CE\w\A5080926\e' for test results files
2023-12-13T18:40:06.788Z INFO run.py __init__(48) read_results Found results file C:\h\w\A22708CE\w\A5080926\e\testResults.xml with format xunit
2023-12-13T18:40:06.788Z INFO run.py __init__(42) read_results Searching 'C:\h\w\A22708CE\w\A5080926\uploads' for test results files
2023-12-13T18:40:06.788Z INFO run.py packing_test_reporter(30) report_results Packing 0 test reports to 'C:\h\w\A22708CE\w\A5080926\e\__test_report.json'
2023-12-13T18:40:06.788Z INFO run.py packing_test_reporter(33) report_results Packed 1537 bytes
ERROR: The process "corerun.exe" not found.
gen-debug-dump-docs.py: Did not find dumps, skipping dump docs generation.
['ComInterfaceGenerator.Tests' END OF WORK ITEM LOG: Command exited with 1]
Known issue validation
Build: :mag_right: https://dev.azure.com/dnceng-public/public/_build/results?buildId=498370
Error message validated: XUnitLogChecker.dll does not exist in the expected location:
Result validation: :white_check_mark: Known issue matched with the provided build.
Validation performed at: 12/14/2023 6:12:51 PM UTC
Report
Summary
| 24-Hour Hit Count | 7-Day Hit Count | 1-Month Count |
|---|---|---|
| 6 | 37 | 141 |
Tagging subscribers to this area: @hoyosjs See info in area-owners.md if you want to be subscribed.
Issue Details
Error Blob
{
"ErrorMessage": "XUnitLogChecker.dll does not exist in the expected location:",
"BuildRetry": false,
"ErrorPattern": "",
"ExcludeConsoleLog": false
}
Reproduction Steps
- Queue:
runtime-libraries-coreclr outerloop - Build example: https://dev.azure.com/dnceng-public/public/_build/results?buildId=498370&view=logs&j=631ed29b-f0dd-5b15-38a0-c6c555556dea&t=b9941189-b5a8-5df3-6b3a-0964fabb3acd
- Log example: https://helixre107v0xdcypoyl9e7f.blob.core.windows.net/dotnet-runtime-refs-pull-95898-merge-d360b025a7c14161a1/ComInterfaceGenerator.Tests/1/console.66454013.log?helixlogtype=result
- Output example:
Console log: 'ComInterfaceGenerator.Tests' from job d360b025-a7c1-4161-a1a4-19f920345fd3 workitem 57966585-bda4-4f37-8ff1-0af071d4373f (windows.81.amd64.open.rt) executed on machine a0018BJ running Windows-2012ServerR2-6.3.9600-SP0
C:\h\w\A22708CE\w\A5080926\e>taskkill.exe /f /im corerun.exe
ERROR: The process "corerun.exe" not found.
C:\h\w\A22708CE\w\A5080926\e>call RunTests.cmd --runtime-path C:\h\w\A22708CE\p
========================= Begin custom configuration settings ==============================
set __TestArchitecture=x64
set __IsXUnitLogCheckerSupported=1
========================== End custom configuration settings ===============================
----- start Wed 12/13/2023 18:40:04.47 =============== To repro directly: =====================================================
pushd C:\h\w\A22708CE\w\A5080926\e\
"C:\h\w\A22708CE\p\dotnet.exe" exec --runtimeconfig ComInterfaceGenerator.Tests.runtimeconfig.json --depsfile ComInterfaceGenerator.Tests.deps.json xunit.console.dll ComInterfaceGenerator.Tests.dll -xml testResults.xml -nologo -nocolor -trait category=OuterLoop -notrait category=IgnoreForCI -notrait category=failing
popd
===========================================================================================================
C:\h\w\A22708CE\w\A5080926\e>"C:\h\w\A22708CE\p\dotnet.exe" exec --runtimeconfig ComInterfaceGenerator.Tests.runtimeconfig.json --depsfile ComInterfaceGenerator.Tests.deps.json xunit.console.dll ComInterfaceGenerator.Tests.dll -xml testResults.xml -nologo -nocolor -trait category=OuterLoop -notrait category=IgnoreForCI -notrait category=failing
Discovering: ComInterfaceGenerator.Tests (method display = ClassAndMethod, method display options = None)
Discovered: ComInterfaceGenerator.Tests (found 0 of 45 test cases)
=== TEST EXECUTION SUMMARY ===
ComInterfaceGenerator.Tests.dll Total: 0
----- end Wed 12/13/2023 18:40:06.22 ----- exit code 0 ----------------------------------------------------------
----- start =============== XUnitLogChecker Output =====================================================
XUnitLogChecker.dll does not exist in the expected location: C:\h\w\A22708CE\p\XUnitLogChecker.dll
----- end =============== XUnitLogChecker Output - exit code 1 ===============
2023-12-13T18:40:06.788Z INFO run.py run(48) main Beginning reading of test results.
2023-12-13T18:40:06.788Z INFO run.py __init__(42) read_results Searching 'C:\h\w\A22708CE\w\A5080926\e' for test results files
2023-12-13T18:40:06.788Z INFO run.py __init__(48) read_results Found results file C:\h\w\A22708CE\w\A5080926\e\testResults.xml with format xunit
2023-12-13T18:40:06.788Z INFO run.py __init__(42) read_results Searching 'C:\h\w\A22708CE\w\A5080926\uploads' for test results files
2023-12-13T18:40:06.788Z INFO run.py packing_test_reporter(30) report_results Packing 0 test reports to 'C:\h\w\A22708CE\w\A5080926\e\__test_report.json'
2023-12-13T18:40:06.788Z INFO run.py packing_test_reporter(33) report_results Packed 1537 bytes
ERROR: The process "corerun.exe" not found.
gen-debug-dump-docs.py: Did not find dumps, skipping dump docs generation.
['ComInterfaceGenerator.Tests' END OF WORK ITEM LOG: Command exited with 1]
| Author: | carlossanlop |
|---|---|
| Assignees: | - |
| Labels: |
|
| Milestone: | - |
Might have to do with how the interop generator test payloads are generated? ComInterfaceGenerator.Tests might be different from other libraries tests.
Has there been any progress on this? This is blocking Known Build Errors on outerloop pipelines because too many test failures are reported in AzDO and legit test failures end up not getting processed by the bot. See discussion on https://github.com/dotnet/runtime/issues/96481#issuecomment-1896427494 for more info.
Tagging subscribers to this area: @dotnet/area-infrastructure-libraries See info in area-owners.md if you want to be subscribed.
Issue Details
Error Blob
{
"ErrorMessage": "XUnitLogChecker.dll does not exist in the expected location:",
"BuildRetry": false,
"ErrorPattern": "",
"ExcludeConsoleLog": false
}
Reproduction Steps
- Queue:
runtime-libraries-coreclr outerloop - Build example: https://dev.azure.com/dnceng-public/public/_build/results?buildId=498370&view=logs&j=631ed29b-f0dd-5b15-38a0-c6c555556dea&t=b9941189-b5a8-5df3-6b3a-0964fabb3acd
- Log example: https://helixre107v0xdcypoyl9e7f.blob.core.windows.net/dotnet-runtime-refs-pull-95898-merge-d360b025a7c14161a1/ComInterfaceGenerator.Tests/1/console.66454013.log?helixlogtype=result
- Output example:
Console log: 'ComInterfaceGenerator.Tests' from job d360b025-a7c1-4161-a1a4-19f920345fd3 workitem 57966585-bda4-4f37-8ff1-0af071d4373f (windows.81.amd64.open.rt) executed on machine a0018BJ running Windows-2012ServerR2-6.3.9600-SP0
C:\h\w\A22708CE\w\A5080926\e>taskkill.exe /f /im corerun.exe
ERROR: The process "corerun.exe" not found.
C:\h\w\A22708CE\w\A5080926\e>call RunTests.cmd --runtime-path C:\h\w\A22708CE\p
========================= Begin custom configuration settings ==============================
set __TestArchitecture=x64
set __IsXUnitLogCheckerSupported=1
========================== End custom configuration settings ===============================
----- start Wed 12/13/2023 18:40:04.47 =============== To repro directly: =====================================================
pushd C:\h\w\A22708CE\w\A5080926\e\
"C:\h\w\A22708CE\p\dotnet.exe" exec --runtimeconfig ComInterfaceGenerator.Tests.runtimeconfig.json --depsfile ComInterfaceGenerator.Tests.deps.json xunit.console.dll ComInterfaceGenerator.Tests.dll -xml testResults.xml -nologo -nocolor -trait category=OuterLoop -notrait category=IgnoreForCI -notrait category=failing
popd
===========================================================================================================
C:\h\w\A22708CE\w\A5080926\e>"C:\h\w\A22708CE\p\dotnet.exe" exec --runtimeconfig ComInterfaceGenerator.Tests.runtimeconfig.json --depsfile ComInterfaceGenerator.Tests.deps.json xunit.console.dll ComInterfaceGenerator.Tests.dll -xml testResults.xml -nologo -nocolor -trait category=OuterLoop -notrait category=IgnoreForCI -notrait category=failing
Discovering: ComInterfaceGenerator.Tests (method display = ClassAndMethod, method display options = None)
Discovered: ComInterfaceGenerator.Tests (found 0 of 45 test cases)
=== TEST EXECUTION SUMMARY ===
ComInterfaceGenerator.Tests.dll Total: 0
----- end Wed 12/13/2023 18:40:06.22 ----- exit code 0 ----------------------------------------------------------
----- start =============== XUnitLogChecker Output =====================================================
XUnitLogChecker.dll does not exist in the expected location: C:\h\w\A22708CE\p\XUnitLogChecker.dll
----- end =============== XUnitLogChecker Output - exit code 1 ===============
2023-12-13T18:40:06.788Z INFO run.py run(48) main Beginning reading of test results.
2023-12-13T18:40:06.788Z INFO run.py __init__(42) read_results Searching 'C:\h\w\A22708CE\w\A5080926\e' for test results files
2023-12-13T18:40:06.788Z INFO run.py __init__(48) read_results Found results file C:\h\w\A22708CE\w\A5080926\e\testResults.xml with format xunit
2023-12-13T18:40:06.788Z INFO run.py __init__(42) read_results Searching 'C:\h\w\A22708CE\w\A5080926\uploads' for test results files
2023-12-13T18:40:06.788Z INFO run.py packing_test_reporter(30) report_results Packing 0 test reports to 'C:\h\w\A22708CE\w\A5080926\e\__test_report.json'
2023-12-13T18:40:06.788Z INFO run.py packing_test_reporter(33) report_results Packed 1537 bytes
ERROR: The process "corerun.exe" not found.
gen-debug-dump-docs.py: Did not find dumps, skipping dump docs generation.
['ComInterfaceGenerator.Tests' END OF WORK ITEM LOG: Command exited with 1]
Known issue validation
Build: :mag_right: https://dev.azure.com/dnceng-public/public/_build/results?buildId=498370
Error message validated: XUnitLogChecker.dll does not exist in the expected location:
Result validation: :white_check_mark: Known issue matched with the provided build.
Validation performed at: 12/14/2023 6:12:51 PM UTC
Report
Summary
| 24-Hour Hit Count | 7-Day Hit Count | 1-Month Count |
|---|---|---|
| 4 | 25 | 92 |
| Author: | carlossanlop |
|---|---|
| Assignees: | - |
| Labels: |
|
| Milestone: | - |
I think I know what's going on...
The regular libraries runs build the product (including XUnitLogChecker) in one job, push it to the feed, and then later it is downloaded and consumed by the job that runs the tests.
In outerloop, we build the product in the same job, in a previous step before the test running step. We aren't building XUnitLogChecker in that build step.
I'm now trying to figure out where and how to add XUnitLogChecker so it builds in outerloop jobs too.
Looks like XunitLogChecker is normally build by libs\tests.proj. This is built in both the regular build and in the outerloop build, however during the outerloop build it seems that the condition to inlcude the checker project is false.
We looked into why this is happening - it's because BuildTargetFramework is not set when the outerloop tests build -- which is happening because the -framework swith is not passed to the build script.
So this is false: https://github.com/dotnet/runtime/blob/b8e1296c832858e7f4b0eb2949afa73147a91981/src/libraries/tests.proj#L759-L763
In outerloop no -framework passed:
https://github.com/dotnet/runtime/blob/b8e1296c832858e7f4b0eb2949afa73147a91981/eng/pipelines/libraries/outerloop.yml#L44
But in the regular libraries build it will be: https://github.com/dotnet/runtime/blob/b8e1296c832858e7f4b0eb2949afa73147a91981/eng/pipelines/libraries/base-job.yml#L74
To fix this we should see if we can just update the condition for including the log checker CSProj to not depend on BuildTargetFramework. Maybe TargetFrameworkIdentifier is sufficient (if we were trying to filter to netcore only).