TokenizerGpt3.Encode incorrect

[3400442579]

Describe the bug TokenizerGpt3.Encode incorrect

Your code piece

 text = """
                Many words map to one token, but some don't: indivisible.

                Unicode characters like emojis may be split into many tokens containing the underlying bytes: 🤚🏾

                Sequences of characters commonly found next to each other may be grouped together: 1234567890
                """;
            int n=TokenizerGpt3.Encode(text).Count;

Result = 260

Screenshots If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: [Windows]
• Language [c#]
• Version [6.7.0]

Additional context Add any other context about the problem here.

[kayhantolga]

That's interesting, I thought I solved this issue in 6.7.0 . Thanks for reporting, I will have a look

[kayhantolga]

solved in v6.7.2