Ryan G.

So in my (limited) experience - the C# version Encode method accepts 1 sentence at a time, I couldn't get it to work with multiple sentences at a time. Ex:...

Not sure if you've experienced this as well, but the Tokenize method also hangs when presented with irregular text. For example, this input will cause the Tokenize method to hang...

Made some improvements to the TokenizeSubwords method in the TokenizerBase to improve general resiliency: `private IEnumerable TokenizeSubwords(string word) { if (this.vocabularyDict.TryGetValue(word, out var count)) { yield return (word, count); yield...