Core tests are flaky

[mrlacey]

trafficstars

Describe the bug

Some of the Minimum Core tests do not produce consistent results. It appears to be that the ones that don't always pass are reliant on the file system.

• [x] Disable unreliable tests
• [ ] Review current tests.
• [ ] remove reliability issues
• [ ] Reenable all tests.

To Reproduce

1. Run Minimum tests a few times
2. See errors

Additional context

Follow on from https://github.com/microsoft/TemplateStudio/pull/4417#issuecomment-1083073959

Applies to the following platforms:

• [X] WinUI
• [X] WPF
• [x] UWP

This potentially applies to all as the impacted tests (and functionality) are part of the "core" logic

About your setup

X-Ref: https://github.com/microsoft/TemplateStudio/pull/4417#issuecomment-1083073959

[mrlacey]

The flaky tests have been disabled for now so as not to block the release of the initial VS2022 extensions. I'm leaving this open and assigned to me to pick up in the future.

[mrlacey]

Flaky tests are reliant on IO as they interact with the filesystem and files are not always written or unlocked as fast as the tests expect. This seems to be more of an issue with VS2022 & on Win11

[mrlacey]

The cause of the IO-related failures is that there are multiple tests that are creating, copying to, and deleting from directories with the same name. When tests are run individually this isn't a problem. However, when tests are run concurrently, this can be a big issue. Even when run serially, this can still be a problem, as there can be very slight delays in the IO operations returning and changes on disk being completed. This can be enough for artifacts from one test to interfere with the next.

As most of the affected tests are inherited from CoreTemplateStudio, I'm reluctant to do any major rewrites of the tests, however, I think a suitable compromise will be to have each test use different directories (based on the test name). This is a small risk of this causing further complications as it will add to path lengths and so could lead to MAX_PATH issues but they can be addressed separately.