Failed to calculate number of tokens with tiktoken

[j1philli]

Hello, I am receiving the following errors when executing my Supabase edge function that is running locally. The function finishes as expected but it would be nice to have these calculations succeed. Any help is appreciated.

Edge Function (sorry for the mess)

import { serve } from "http/server.ts";
import { ChatOpenAI } from "https://esm.sh/langchain/chat_models/openai";
import { OpenAIEmbeddings } from "https://esm.sh/langchain/embeddings/openai";
import { LLMChain } from "https://esm.sh/langchain/chains";
import { CallbackManager } from "https://esm.sh/langchain/callbacks";
import {
  ChatPromptTemplate,
  HumanMessagePromptTemplate,
} from "https://esm.sh/langchain/prompts";
import { createClient } from "https://esm.sh/@supabase/[email protected]";
import { SupabaseHybridSearch } from "https://esm.sh/langchain/retrievers/supabase";
import { SupabaseVectorStore } from "https://esm.sh/langchain/vectorstores/supabase";
import { OpenAI } from "https://esm.sh/langchain/llms/openai";
import { loadQAStuffChain, loadQAMapReduceChain } from "https://esm.sh/langchain/chains";
import { Document } from "https://esm.sh/langchain/document";

import { corsHeaders } from "../_shared/cors.ts";

const openaikey = "sk-xxx";
const prompt = ChatPromptTemplate.fromPromptMessages([
  HumanMessagePromptTemplate.fromTemplate("{input}"),
]);

serve(async (req) => {

  const supabase = createClient(
    Deno.env.get('SUPABASE_URL') ?? '',
    Deno.env.get('SUPABASE_SERVICE_ROLE_KEY') ?? '',
    { global: { headers: { Authorization: req.headers.get('Authorization')! } } }
  );
  if (!supabase) throw new Error('supabase client was not created as expected')
  console.log("client created")

  // This is needed if you're planning to invoke your function from a browser.
  if (req.method === "OPTIONS") {
    return new Response("ok", { headers: corsHeaders });
  }
  const { input } = await req.json();

  const embeddings = new OpenAIEmbeddings({openAIApiKey: openaikey});
  console.log("embeddings is created")
  const retriever = new SupabaseHybridSearch(embeddings, {
    client: supabase,
    //  Below are the defaults, expecting that you set up your supabase table and functions according to the guide above. Please change if necessary.
    similarityK: 2,
    keywordK: 2,
    tableName: "documents",
    similarityQueryName: "match_documents",
    keywordQueryName: "kw_match_documents",
  });
  console.log("retriever is created")
  console.log(JSON.stringify({ input }))
  
  const docs = await retriever.getRelevantDocuments(JSON.stringify({ input }));
  console.log("response is created")

  const llmB = new OpenAI({openAIApiKey: openaikey, maxConcurrency: 10 });
  const chainB = loadQAMapReduceChain(llmB);
  const resB = await chainB.call({
    input_documents: docs,
    question: JSON.stringify({ input }),
});
  return new Response(JSON.stringify(resB), {
  headers: { ...corsHeaders, "Content-Type": "application/json" },
  });
});

Console Error

response is created
Failed to calculate number of tokens with tiktoken, falling back to approximate count TypeError: I.__wbindgen_add_to_stack_pointer is not a function
    at q (https://esm.sh/v117/@dqbd/[email protected]/deno/tiktoken.mjs:2:3977677)
    at ee.getNumTokens (https://esm.sh/v117/[email protected]/deno/llms/openai.js:7:18182)
    at async Promise.all (index 0)
    at async O._call (https://esm.sh/v117/[email protected]/deno/chains.js:14:1932)
    at async O.call (https://esm.sh/v117/[email protected]/deno/chains.js:169:4127)
    at async Server.<anonymous> (file:///home/deno/functions/aifetcher/index.ts:63:16)
    at async Server.#respond (https://deno.land/[email protected]/http/server.ts:299:18)
Failed to calculate number of tokens with tiktoken, falling back to approximate count TypeError: I.__wbindgen_add_to_stack_pointer is not a function
    at q (https://esm.sh/v117/@dqbd/[email protected]/deno/tiktoken.mjs:2:3977677)
    at ee.getNumTokens (https://esm.sh/v117/[email protected]/deno/llms/openai.js:7:18182)
    at async Promise.all (index 1)
    at async O._call (https://esm.sh/v117/[email protected]/deno/chains.js:14:1932)
    at async O.call (https://esm.sh/v117/[email protected]/deno/chains.js:169:4127)
    at async Server.<anonymous> (file:///home/deno/functions/aifetcher/index.ts:63:16)
    at async Server.#respond (https://deno.land/[email protected]/http/server.ts:299:18)

[Anil-matcha]

Any update, I am facing same error in browser

__wbindgen_add_to_stack_pointer is not a function

[j1philli]

Nothing yet from me. I've moved on from investigating this for now. Hoping the community is able to figure it out.

[aseem2625]

Hey @hwchase17 possible to look into this on?

This error cannot be catched and is ruining the logs due to internal retries.

I am porting ConversationTokenBufferMemory to JS library but this is some blocker.

[logancyang]

I'm also seeing this error in the browser. What I'm doing is just

const chatOpenAI = new ChatOpenAI({
  openAIApiKey: key,
  modelName: 'gpt-3.5-turbo',
  temperature: 0.7,
  maxTokens: 1000,
  streaming: true,
});

chatOpenAI.getNumTokens("some text i pass in");

Error logged as warning message:

Failed to calculate number of tokens with tiktoken, falling back to approximate count TypeError: wasm.__wbindgen_add_to_stack_pointer is not a function
    at encoding_for_model (tiktoken_bg.js:199:14)
    at ChatOpenAI.getNumTokens

Would appreciate any help and insight! @dqbd

[dqbd]

Hello @logancyang!

This issue shoud be fixed with by upgrading LangchainJS to 0.0.76 or newer, which replaces the WASM tiktoken package with a pure-JS version.

[logancyang]

@dqbd thank you so much 🙏 Just upgraded and it works!

[j1philli]

My logs aren't full of the errors anymore! Now to figure out how to actually use tiktoken. Thanks @dqbd