Tiktoken package for openai script not working

[azmathr]

In my pthon script I have tiktoken enabled which essencially converts characters to openai tokens. Its a common module used in openai. Here is an associated code.

encoding = tiktoken.get_encoding("cl100k_base")

`def num_tokens_from_string(string: str, encoding_name: str) -> int: """Returns the number of tokens in a text string.""" encoding = tiktoken.get_encoding(encoding_name) num_tokens = len(encoding.encode(string)) return num_tokens

def get_text_until_limit(elements, limit): text = '' for element in elements: new_text = text + ' ' + element.get_text() if num_tokens_from_string(new_text, "cl100k_base") <= limit: text = new_text else: break return text ` Unfortunately this is not working through zappa. I am getting error

 encoding = tiktoken.get_encoding("cl100k_base")   File "/var/task/tiktoken/registry.py", line 64, in get_encoding     _find_constructors()   File "/var/task/tiktoken/registry.py", line 44, in _find_constructors     raise ValueError( [1707923842317] [DEBUG] 2024-02-1

warmhandler error