Token Type Analyzer: use Razor Tokenizer for razor files

[antonioaversa]

In the current approach (not yet merged to master), the token type analyzer currently maps back tokens from the generated source to the razor source.

This approach is an effective first baby step to provide quick support for Razor syntax highlighting, but comes with some limitations:

• depends on location mapping, which is based on line directives and affected by several bugs
• only works for C# tokens: HTML tokens and Razor tokens (e.g. @using, @page, @* etc.) are not supported

The alternative, and the goal of this task, is to have a Razor-specific implementation of the TokenTypeAnalyzer that uses RazorSyntaxTree.

See razor unit tests on tokenization (some examples here) for some examples of its use.

That would generate all types of tokens, since Razor tokens embed C# ones: see here.

Disadvantages:

• requires new parsing (or at least tokenization - can we have the second without paying the cost of full parsing? Probably not...)
• requires maintenance every time a new version of C# and/or Razor is released, that introduces new tokens