openai
openai copied to clipboard
TokenizerGpt3.Encode incorrect
Describe the bug TokenizerGpt3.Encode incorrect
Your code piece
text = """
Many words map to one token, but some don't: indivisible.
Unicode characters like emojis may be split into many tokens containing the underlying bytes: 🤚🏾
Sequences of characters commonly found next to each other may be grouped together: 1234567890
""";
int n=TokenizerGpt3.Encode(text).Count;
Result = 260
Screenshots
If applicable, add screenshots to help explain your problem.
Desktop (please complete the following information):
- OS: [Windows]
- Language [c#]
- Version [6.7.0]
Additional context Add any other context about the problem here.
That's interesting, I thought I solved this issue in 6.7.0 . Thanks for reporting, I will have a look
solved in v6.7.2