MSTest performance

[Youssef1313]

This issue tracks few ideas for improving the performance of MSTest.

• [ ] We shouldn't do any serialization for parameterized tests if appdomain is disabled. This will be v4 only where we changed the test id to not rely serialized data.

• [x] ManagedNameHelper from VSTest is extremely slow and memory consuming (PR: https://github.com/microsoft/testfx/pull/6319)

  • Note: on a single test class with 10k tests.

    

• [x] Dictionary<string, string> used in UpdateInfoIfTestInitializeOrCleanupMethod:

  • is used to act more as of HashSet. So arguably a HashSet will be simply faster.
  • is consuming more time and memory for providing zero functionality if the test class inherits directly from object.
  • Its overall value is too low (prevents considering base initializes if they are shadowed in concrete test class). Should we get rid of it altogether?
  • [x] Switch to HashSet and avoiding it altogether when base type is object is in PR https://github.com/microsoft/testfx/pull/6318. Removing the whole thing as "low value" is still an open question.

[Youssef1313]

Quick profiling (single run, not averaging, but makes it quite obvious 3.11 is significantly faster in multiple scenarios due to #6319)

Single test class with 10k tests

3.3.1: 10s 3.4.3: 14s 3.5.2: 9s 3.6.4: 9s 3.7.3: 10s 3.8.3: 15s 3.9.3: 9s 3.10.1: 8s 3.11 pre: 3s

1k test class, each with 200 tests

3.3.1: 28s 3.4.3: 29s 3.5.2: 27s 3.6.4: 25s 3.7.3: 28s 3.8.3: 39s 3.9.3: 26s 3.10.1: 33s 3.11 pre: 20s

100 test class, each with 2000 tests

3.3.1: 43s 3.4.3: 49s 3.5.2: 33s 3.6.4: 32s 3.7.3: 44s 3.8.3: 42s 3.9.3: 43s 3.10.1: 37s 3.11 pre: 17s

[Youssef1313]

I did two extra runs. So here is the result of 3 runs of each scenario:

Single test class with 10k tests

3.3.1: 10s 10s 9s 3.4.3: 14s 9s 10s 3.5.2: 9s 9s 9s 3.6.4: 9s 10s 10s 3.7.3: 10s 9s 8s 3.8.3: 15s 8s 9s 3.9.3: 9s 9s 9s 3.10.1: 8s 8s 9s 3.11 pre: 3s 2s 3s

1k test class, each with 200 tests

3.3.1: 28s 29s 29s 3.4.3: 29s 29s 27s 3.5.2: 27s 27s 27s 3.6.4: 25s 25s 27s 3.7.3: 28s 24s 25s 3.8.3: 39s 28s 26s 3.9.3: 26s 25s 25s 3.10.1: 33s 21s 23s 3.11 pre: 20s 17s 18s

100 test class, each with 2000 tests

3.3.1: 43s 44s 42s 3.4.3: 49s 41s 42s 3.5.2: 33s 33s 33s 3.6.4: 32s 32s 32s 3.7.3: 44s 41s 40s 3.8.3: 42s 43s 43s 3.9.3: 43s 42s 42s 3.10.1: 37s 40s 39s 3.11 pre: 17s 18s 18s

[nohwnd]

graphed by copilot, since my brain does not comprehend the numbers without being in a table.