Remove allocations on all base converters, improve TokenizerHelper

[h3xds1nz]

Description

This is probably one of the impactful PRs I've posted; optimizes conversion/parsing of all the base-generated types but also ThicknessConverter (once more), CornerRadiusConverter, KeySplineConverter and VirtualizationCacheLengthConverter.

Completely removes any allocations caused by the original Tokenizer by using ReadOnlySpan<char> instead of substrings. I have also introduced a ref struct variant that can be used in any code that doesn't compile for net472.

MilCodeGen has also been adjusted to support this.

Single Thickness parsing (4 units)

Method	Mean [ns]	Error [ns]	StdDev [ns]	Code Size [B]	Gen0	Allocated [B]
Original	237.4 ns	4.64 ns	4.34 ns	555 B	0.0105	176 B
PR_EDITREF	191.6 ns	1.62 ns	1.51 ns	761 B	-	-

Parsing Thickness 10 times (4 units)

Method	Mean [ns]	Error [ns]	StdDev [ns]	Code Size [B]	Gen0	Allocated [B]
Original	2,434.0 ns	17.02 ns	14.08 ns	608 B	0.1030	1760 B
PR_EDITREF	1,885.1 ns	5.68 ns	4.44 ns	814 B	-	-

Creation of 10x Matrix3D via Parse

Method	Mean [ns]	Error [ns]	StdDev [ns]	Gen0	Code Size [B]	Allocated [B]
Original	7,643.3 ns	26.76 ns	20.89 ns	0.3052	6,876 B	5200 B
PR_EDIT	5,885.4 ns	67.98 ns	60.27 ns	-	7,177 B	-

Customer Impact

Improved performance by around 20% on base types, zeroed out allocations.

Regression

No.

Testing

Local build, basic testing of the tokenizer with span and some of the converters.

Risk

Low, the scope of the PR is big but with a simple change, hence any mistakes shall be easy to figure out on a DRT run.

Microsoft Reviewers: Open in CodeFlow

[h3xds1nz]

Resolved merge conflicts 12, there are better ways to spent an hour and a half.

[h3xds1nz]

@himgoyalmicro I've noticed this has been cut, can you share details about the failed test(s)?

This is really important PR and preferrably it should be in .NET 10.

[himgoyalmicro]

Hey @h3xds1nz, Sorry for the delay in response. We have observed that when we do Rect testRect = Rect.Parse(" Empty \t"); We are hitting the exception System.FormatException: 'The input string 'Empty' was not in correct format.'

[h3xds1nz]

@himgoyalmicro Right, that was silly. Fixed.

~I had to include #10683 for now to be able to run MilCodeGen, that's a PR that doesn't really need tests, could be just approved.~

[codecov[bot]]

Codecov Report

Attention: Patch coverage is 45.36341% with 218 lines in your changes missing coverage. Please review.

Project coverage is 11.41570%. Comparing base (3cc9d57) to head (8b733a1). Report is 64 commits behind head on main.

Additional details and impacted files

@@                 Coverage Diff                 @@
##                main       #9364         +/-   ##
===================================================
+ Coverage   11.17950%   11.41570%   +0.23619%     
===================================================
  Files           3314        3316          +2     
  Lines         665182      665312        +130     
  Branches       74668       74690         +22     
===================================================
+ Hits           74364       75950       +1586     
+ Misses        589525      587950       -1575     
- Partials        1293        1412        +119     

Flag	Coverage Δ
Debug	11.41570% <45.36341%> (+0.23619%)	:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

:rocket: New features to boost your workflow:

• :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.