NewlyCreatedProjectsFromDotNetNew.ValidateCSharpTemplateProjects is flakey

[CyrusNajmabadi]

Failed https://dev.azure.com/dnceng-public/public/_build/results?buildId=720157&view=ms.vss-test-web.build-test-results-tab&runId=18060982&resultId=100226&paneView=debug

Expected 0 items but found 17: The following compiler diagnostics are being reported for the template.
Items:
T:\RoslynTests\65921193-f413-4206-afc8-9657782aa1d4\obj\Debug\net9.0\65921193-f413-4206-afc8-9657782aa1d4.GlobalUsings.g.cs(2,32): error CS0234: The type or namespace name 'Playwright' does not exist in the namespace 'Microsoft' (are you missing an assembly reference?)
T:\RoslynTests\65921193-f413-4206-afc8-9657782aa1d4\obj\Debug\net9.0\65921193-f413-4206-afc8-9657782aa1d4.GlobalUsings.g.cs(3,22): error CS0400: The type or namespace name 'NUnit' could not be found in the global namespace (are you missing an assembly reference?)
T:\RoslynTests\65921193-f413-4206-afc8-9657782aa1d4\UnitTest1.cs(5,22): error CS0246: The type or namespace name 'PageTest' could not be found (are you missing a using directive or an assembly reference?)
T:\RoslynTests\65921193-f413-4206-afc8-9657782aa1d4\UnitTest1.cs(3,2): error CS0246: The type or namespace name 'ParallelizableAttribute' could not be found (are you missing a using directive or an assembly reference?)
T:\RoslynTests\65921193-f413-4206-afc8-9657782aa1d4\UnitTest1.cs(3,2): error CS0246: The type or namespace name 'Parallelizable' could not be found (are you missing a using directive or an assembly reference?)
T:\RoslynTests\65921193-f413-4206-afc8-9657782aa1d4\UnitTest1.cs(4,2): error CS0246: The type or namespace name 'TestFixtureAttribute' could not be found (are you missing a using directive or an assembly reference?)
T:\RoslynTests\65921193-f413-4206-afc8-9657782aa1d4\UnitTest1.cs(4,2): error CS0246: The type or namespace name 'TestFixture' could not be found (are you missing a using directive or an assembly reference?)
T:\RoslynTests\65921193-f413-4206-afc8-9657782aa1d4\UnitTest1.cs(3,17): error CS0103: The name 'ParallelScope' does not exist in the current context
T:\RoslynTests\65921193-f413-4206-afc8-9657782aa1d4\UnitTest1.cs(7,6): error CS0246: The type or namespace name 'TestAttribute' could not be found (are you missing a using directive or an assembly reference?)
T:\RoslynTests\65921193-f413-4206-afc8-9657782aa1d4\UnitTest1.cs(7,6): error CS0246: The type or namespace name 'Test' could not be found (are you missing a using directive or an assembly reference?)
T:\RoslynTests\65921193-f413-4206-afc8-9657782aa1d4\UnitTest1.cs(10,15): error CS0103: The name 'Page' does not exist in the current context
T:\RoslynTests\65921193-f413-4206-afc8-9657782aa1d4\UnitTest1.cs(13,15): error CS0103: The name 'Expect' does not exist in the current context
T:\RoslynTests\65921193-f413-4206-afc8-9657782aa1d4\UnitTest1.cs(13,22): error CS0103: The name 'Page' does not exist in the current context
T:\RoslynTests\65921193-f413-4206-afc8-9657782aa1d4\UnitTest1.cs(16,26): error CS0103: The name 'Page' does not exist in the current context
T:\RoslynTests\65921193-f413-4206-afc8-9657782aa1d4\UnitTest1.cs(19,15): error CS0103: The name 'Expect' does not exist in the current context
T:\RoslynTests\65921193-f413-4206-afc8-9657782aa1d4\UnitTest1.cs(25,15): error CS0103: The name 'Expect' does not exist in the current context
T:\RoslynTests\65921193-f413-4206-afc8-9657782aa1d4\UnitTest1.cs(25,22): error CS0103: The name 'Page' does not exist in the current context

[ghost]

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[ghost]

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[RikkiGibson]

@jjonescz I seem to recall you filing a bug on a template author recently? Is this issue related to that?

[jjonescz]

I seem to recall you filing a bug on a template author recently? Is this issue related to that?

Yes: https://github.com/dotnet/test-templates/issues/412 I have even disabled the failing playwright test scenario in https://github.com/dotnet/roslyn/pull/73656/commits/9acbcf77dd9351ea0bff8707e8214f1b4e19fd1d but perhaps that stopped working because I see the whole theory is now disabled against this issue

https://github.com/dotnet/roslyn/blob/a6824ae36db3352538fd4892dbb11325c73d40d0/src/Workspaces/MSBuildTest/NewlyCreatedProjectsFromDotNetNew.cs#L64

[jjonescz]

Anyway that bug should be fixed in .NET 9 preview 6 which should flow in through arcade soon I think.

[RikkiGibson]

The CI build where the problem occurred was deleted. Let's keep an eye out to see if this recurs but otherwise closing as unactionable/likely solved by external changes.

[jjonescz]

Let's keep an eye out to see if this recurs but otherwise closing as unactionable/likely solved by external changes.

The test is currently disabled with this issue linked. We would need to re-enable to see if it's fixed.

[JoeRobich]

The skipped test has been reenabled. Closing