Fix tokens returning unicode partial chars

[VansonLeung]

Fix to prevent seeing � characters when chatting in Chinese.

[AlexCheema]

Is this how other APIs fix this? Or is this typically handled by the consumer of the API?

[VansonLeung]

I figured out the fix myself, I neither have no clue how other APIs fix this nor I am aware that other APIs have this issue as well.

Some unicode / Chinese characters consist of 2 integers/chars in order to render properly.

My fix hopes to avoid changing any API logic.

[Evanev7]

Thanks so much for your contribution and for taking the time to open this PR.

Since this repository has been fully rewritten and the license has changed, I’m closing all existing open PRs to avoid confusion and to align with the new codebase.

I really appreciate your interest in the project, and you’re very welcome to open a new PR against the updated version if you’d like and we look forward to reviewing it!