As you may know, function calls are now supported by OpenAI, and the function call tokens will be taken into account.
With some googling, I found the calculation here.
It will be great if the library could also calculate the required tokens when using with function calls. 🙏🏼
gpt-tokenizer copied to clipboard
niieani
