Tokenize<TKind>.Tokenize should accept TextSpan

[sebastienros]

I am pre-parsing some text to extract the parts I need to parse, and don't want to allocate more than necessary. For this reason it would be nice to be able to pass a TextSpan that I would extract from the original buffer instead of having to create a string out of it and pass it to the tokenizer.

[sebastienros]

Mitigation right now is to copy-paste the protected implementation in my custom tokenizer.

[nblumhardt]

That's interesting - thanks for the note 👍

[nblumhardt]

The challenge here is going to be that Tokenize(TextSpan) already exists in Tokenizer but has a different return type and semantics :-/ ... I'd guess we'll find some way around it, I'll give it some thought.