Tokenizer icon indicating copy to clipboard operation
Tokenizer copied to clipboard

Add token counting tokens to ITokenizer and TikTokenizer

Open pgroene opened this issue 10 months ago • 0 comments

Updated the ITokenizer interface to include two new methods for counting tokens in a string, with options for handling special tokens. Implemented these methods in the TikTokenizer class, adding logic for counting based on special tokens and a maximum limit. Added unit tests in TokenizerTest to verify the new counting functionality, including cases for special tokens and empty strings. In case of checking length the allocation of the list is not done. When using the max count value, the method stops counting when the maxinum number of tokens is exceeded. This prevents endless counting when extreemly large content is provided.

pgroene avatar Feb 17 '25 19:02 pgroene