Intermittent CI failures with the "Build Bicep In Visual Studio" step

[anthony-c-martin]

Example run: https://github.com/Azure/bicep/runs/7561061310

The "Run bicep in visual studio integration tests" step fails with the following output

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
Test Run deployment issue: Failed to get the file for deployment item 'Context.xml': System.IO.FileNotFoundException: Could not find file 'D:\a\bicep\bicep\src\vs-bicep\Bicep.VSLanguageServerClient.IntegrationTests\bin\Release\net472\Context.xml'.
Test Run deployment issue: Failed to get the file for deployment item 'TestcaseInfo.runproj': System.IO.FileNotFoundException: Could not find file 'D:\a\bicep\bicep\src\vs-bicep\Bicep.VSLanguageServerClient.IntegrationTests\bin\Release\net472\TestcaseInfo.runproj'.
Test Run deployment issue: Failed to get the file for deployment item 'Microsoft.Test.Apex.ElevateClient.exe': System.IO.FileNotFoundException: Could not find file 'D:\a\bicep\bicep\src\vs-bicep\Bicep.VSLanguageServerClient.IntegrationTests\bin\Release\net472\Microsoft.Test.Apex.ElevateClient.exe'.
Test Run deployment issue: Failed to get the file for deployment item 'Microsoft.Internal.Performance.CodeMarkers.dll': System.IO.FileNotFoundException: Could not find file 'D:\a\bicep\bicep\src\vs-bicep\Bicep.VSLanguageServerClient.IntegrationTests\bin\Release\net472\Microsoft.Internal.Performance.CodeMarkers.dll'.
Test Run deployment issue: Failed to get the file for deployment item 'x64\Microsoft.Internal.Performance.CodeMarkers.dll' (output directory 'x64'): System.IO.FileNotFoundException: Could not find file 'D:\a\bicep\bicep\src\vs-bicep\Bicep.VSLanguageServerClient.IntegrationTests\bin\Release\net472\Microsoft.Internal.Performance.CodeMarkers.dll'.
  Passed Verify_Classifications [7 m 39 s]
  Passed Verify_TopLevelPlainTextDeclarations [11 s]
  Passed Verify_Formatting [5 s]
  Passed Verify_HoverInformation [5 s]
  Failed Verify_Diagnostics [6 s]
  Error Message:
   Assert.AreEqual failed. Expected:<3>. Actual:<0>. 
  Stack Trace:
     at Bicep.VSLanguageServerClient.IntegrationTests.ValidationTests.Verify_Diagnostics() in D:\a\bicep\bicep\src\vs-bicep\Bicep.VSLanguageServerClient.IntegrationTests\ValidationTests.cs:line 31

  Standard Output Messages:
 

Debug Trace:
Assembly 'D:\a\bicep\bicep\TestResults\Deploy_runneradmin 2022-07-28 13_49_13\Out\Microsoft.Test.Apex.VisualStudio.dll' added to 'VisualStudioHostConfiguration' as 'Single'
 Assembly 'D:\a\bicep\bicep\TestResults\Deploy_runneradmin 2022-07-28 13_49_13\Out\Microsoft.Test.Apex.Framework.dll' added to 'VisualStudioHostConfiguration' as 'Single'
 Assembly 'D:\a\bicep\bicep\TestResults\Deploy_runneradmin 2022-07-28 13_49_13\Out\Microsoft.Test.Apex.VisualStudio.dll' added to 'VisualStudioHostConfiguration' as 'Single'
 Assembly 'D:\a\bicep\bicep\TestResults\Deploy_runneradmin 2022-07-28 13_49_13\Out\Microsoft.Test.Apex.VisualStudio.dll' added to 'VisualStudioHostConfiguration' as 'Single'
 Assembly 'D:\a\bicep\bicep\TestResults\Deploy_runneradmin 2022-07-28 13_49_13\Out\Microsoft.Test.Apex.Framework.dll' added to 'VisualStudioHostConfiguration' as 'Single'
 Assembly 'D:\a\bicep\bicep\TestResults\Deploy_runneradmin 2022-07-28 13_49_13\Out\Microsoft.Test.Apex.VisualStudio.dll' added to 'VisualStudioHostConfiguration' as 'Single'
 

TestContext Messages:
+ ValidationTests::Verify_Diagnostics
     + ApexTest TestInitialize
     + ExternalProcessHostTest TestInitialize
     Error: (5:26.396) [Platform:Testcase] Bicep.VSLanguageServerClient.IntegrationTests.ValidationTests.Verify_Diagnostics failed due to an exception, but the stack trace was unavailable. Refer to the mstest log if necessary.
     + ExternalProcessHostTest TestCleanup
         (5:26.836) [Platform:Testcase] Test 'Verify_Diagnostics' failed. The state of the 'Visual Studio Host (816)' with process '816' is indeterminate due to the failure. Therefore, Apex will shut down the associated host and process. A subsequent test scenario will run on a new instance of the host.
         Warning: (5:26.846) [Platform:TestLoggerDefault] Unable to retrieve an "IVsResourceManagerCacheControl".
         Warning: (5:27.029) [Platform:VisualStudioHost] Ignoring exception encountered while trying to execute Window.ResetWindowLayout:Unable to get service of type 'EnvDTE.IVsExtensibility' as 'Microsoft.VisualStudio.Shell.Interop.IVsExtensibility3' from host
         Warning: (5:27.029) [Platform:VisualStudioHost] Microsoft.Test.Apex.HostServiceException: Unable to get service of type 'EnvDTE.IVsExtensibility' as 'Microsoft.VisualStudio.Shell.Interop.IVsExtensibility3' from host
    at Microsoft.Test.Apex.VisualStudio.Services.VisualStudioCommandService.EnterAutomationFunction()
    at Microsoft.Test.Apex.VisualStudio.VisualStudioUIHostBase.ResetVisualStudioStateWithDte()
         (5:28.186) [Platform:RoslynConfigLifetimeAction] Reset solution crawler after the test...
         (5:28.187) [Platform:RoslynConfigLifetimeAction] Executing 'C:\Program Files\Microsoft Visual Studio\2022\Enterprise\Common7\IDE\VsRegEdit.exe remove "C:\Program Files\Microsoft Visual Studio\2022\Enterprise" HKCU Roslyn\Internal\SolutionCrawler "Solution Crawler"
         (5:28.349) [Platform:RoslynConfigLifetimeAction] Output: Removed value Solution Crawler.
 
         (5:28.354) [Platform:Testcase] The host with process id '816' was shutdown.
     + Final Test Result Verification
         (5:28.361) [Platform:DelayedAssertionVerifierSink] Verification Success: 0 verification steps failed.
     + ApexTest TestCleanup

[anthony-c-martin]

@bhsubra - please could you take a look?

[bhsubra]

Checked in a change to stabilize this test: https://github.com/Azure/bicep/pull/7744

I'll watch the runs.

[anthony-c-martin]

Thanks!

[StephenWeatherford]

Bhavya says is pretty stable now.

[anthony-c-martin]

@StephenWeatherford - note that the outcome of 2 steps are currently being ignored - see https://github.com/Azure/bicep/issues/8078.

[StephenWeatherford]

cc @bhsubra