Change Token to be a struct

[lahma]

Work started here, let's see if it flies with performance benefits: https://github.com/sebastienros/esprima-dotnet/pull/236

[lahma]

Now that JsxToken inherits from Token which would have been struct things are not as straightforward.

[adams85]

Though inheritance is not possible, we still have composition as an option:

public struct Token
{
    public TokenType Type;
    // ...

    public object? ExtensionData;
}`

Of course, this only solves the base parser's problem, JsxParser can't really avoid allocations. However, JsxToken needs an extra field only, which has two possible values (see JsxTokenType). These can be pre-cached in static readonly fields:

private static readonly object JsxIdentifierTokenType = JsxTokenType.JsxIdentifier;
private static readonly object JsxTextTokenType = JsxTokenType.JsxText;

And then

var token = new JsxToken()
{
    Type = JsxTokenType.JsxText,
    Value = text,
    // ...
};

becomes:

var token = new Token()
{
    Type = TokenType.Extension,
    ExtensionData = JsxTextTokenType 
    Value = text,
    // ...
};