Tokenizer
Tokenizer copied to clipboard
Add token counting tokens to ITokenizer and TikTokenizer
Updated the ITokenizer interface to include two new methods for counting tokens in a string, with options for handling special tokens. Implemented these methods in the TikTokenizer class, adding logic for counting based on special tokens and a maximum limit. Added unit tests in TokenizerTest to verify the new counting functionality, including cases for special tokens and empty strings. In case of checking length the allocation of the list is not done. When using the max count value, the method stops counting when the maxinum number of tokens is exceeded. This prevents endless counting when extreemly large content is provided.