Token Type Analyzer testing: improve position asserting for Razor

[antonioaversa]

Unlike C#, razor files have three different types of tokens, all mapped here:

• HTML Tokens, roughly corresponding to the tokens in the "markup" and "HTML" section
• Razor Tokens, such as document, blocks, directives but also Razor comments, etc.
• C# Tokens

Notice that Microsoft.AspNetCore.Razor.Language.SyntaxKind doesn't extend Microsoft.CodeAnalysis.CSharp.SyntaxKind: the two enums are not compatible with each other.

The token position asserting via [x:...] syntax works well for cs files since C# are easy to recognize even when the file is made syntactically invalid, due to the introduction of [x:...] syntax, so code coloring in Visual Studio would stop working.

To overcome this issue and support easy token assertions on large razor files, we could change the way in which token positions are asserted:

• odd lines would be code lines
• even lines would be assertion lines, which would work as squiggles in issues assertions ("^^^..."), but using the acronym of the token type instead of the character ^

For example:

<p id=@aStaticField>@nameof(int.MaxValue)@typeof(int).ToString("G")</p>
                     kkkkkk ttt           kkkkkk ttt           sss

Very few even lines would be empty, since most lines contain at least a token to be asserted. Moreover, tokens don't ever overlap, so the assertion format should be consistent.

Limitation of the proposal

Such format would preserve syntactical validity when asserting Razor lines, because even lines in that format would be interpreted as simple HTML text. It would not, however, within multiline code blocks, such as @code { }, @{ } and similar.

@{
                                  <- this line is empty (no C# tokens on the line above)
  public static void Method() {}
  kkkkkk kkkkkk kkkk              <- this line is not syntactically valid, because in C# code block
}
                                  <- this line is empty (no C# tokens on the line above)

One way of fixing the issue would be by having even lines after C# lines starting with // That means, however, that we could not assert tokens placed on the first two chars of the line. Or in alternative, we would have everything displaced of two chars.

@{

  public static void Method() {}
//kkkkkk kkkkkk kkkk <- alternative 1
//  kkkkkk kkkkkk kkkk <- alternative 2 (misaligned)
}