Fast Up-to-Date Check tests produce different output with `dotnet test`

[tmeschter]

I'm looking into moving our unit tests over to use dotnet test instead of xunit.runner.console.x86.exe and simplifying some of our unit testing infrastructure along the way. This has led to several unit test failures that appear to be due mostly to unstable output:

• IsUpToDateAsync_False_Kinds_InputNewerThanOutput_NoKindIgnored
• IsUpToDateAsync_False_Kinds_InputNewerThanOutput_WithIgnoredKind
• IsUpToDateAsync_True_Kinds_InputNewerThanOutput_NoKindIgnored
• IsUpToDateAsync_True_Kinds_InputNewerThanOutput_WithIgnoredKind

For at least one of these, however, the output has differences beyond ordering issues. For example, in IsUpToDateAsync_False_Kinds_InputNewerThanOutput_NoKindIgnored we see:

Expected:
Adding UpToDateCheckOutput outputs:
    C:\Dev\Solution\Project\TaggedOutput
    C:\Dev\Solution\Project\Output
Adding UpToDateCheckBuilt outputs:
    C:\Dev\Solution\Project\TaggedBuilt
    C:\Dev\Solution\Project\Built
Adding project file inputs:
    C:\Dev\Solution\Project\Project.csproj
Adding newest import input:
    C:\Dev\Solution\Project\Project.csproj
Adding UpToDateCheckInput inputs:
    C:\Dev\Solution\Project\TaggedInput
    C:\Dev\Solution\Project\Input
Input UpToDateCheckInput item 'C:\Dev\Solution\Project\Input' is newer (2022-10-26 14:21:34.513) than earliest output 'C:\Dev\Solution\Project\Output' (2022-10-26 14:19:34.513), not up-to-date.

Actual:
Adding UpToDateCheckOutput outputs:
    C:\Dev\Solution\Project\Output
    C:\Dev\Solution\Project\TaggedOutput
Adding UpToDateCheckBuilt outputs:
    C:\Dev\Solution\Project\Built
    C:\Dev\Solution\Project\TaggedBuilt
Adding project file inputs:
    C:\Dev\Solution\Project\Project.csproj
Adding newest import input:
    C:\Dev\Solution\Project\Project.csproj
Adding UpToDateCheckInput inputs:
    C:\Dev\Solution\Project\Input
Input UpToDateCheckInput item 'C:\Dev\Solution\Project\Input' is newer (2022-10-26 14:21:34.513) than earliest output 'C:\Dev\Solution\Project\Output' (2022-10-26 14:19:34.513), not up-to-date.

Note the difference in the UpToDateCheckInput inputs.

I'm going to disable these tests in order to move forward with the update.

[drewnoakes]

For at least one of these, however, the output has differences beyond ordering issues. 

This is still just ordering. The check bails on the first failure, and with the reordering the failure comes sooner, resulting in a different number of log messages.

[drewnoakes]

Re-opening as these tests were suppressed again in e80814f9708936ac899931abcb85ccd62d7f82ae.

@tmeschter do you have any information on why these are failing again? I think the order should be deterministic again given https://github.com/dotnet/project-system/pull/8645. Could it be that they're actually fine and that the reintroduced skips were based on an old PR? Assigning to you and sending back to triage.

cc @smitpatel