azmathr

Since my file was large GPT would timeout with GPTSimpleVectorIndex. So I split the document into multiple files and saved though GPTSimpleVectorIndex. After slitting I merged them using GPTListIndex. But...

In my pthon script I have tiktoken enabled which essencially converts characters to openai tokens. Its a common module used in openai. Here is an associated code. `encoding = tiktoken.get_encoding("cl100k_base")...