Error with @dqbd/tiktoken when using RecursiveCharacterTextSplitter in langchain on AWS Lambda with Node.js 18.x runtime

[johna1203]

Hi all,

I'm using langchain on an AWS Lambda with Node.js runtime version NODEJS_18_X. I'm trying to create a summary of an array of documents, and I'm using RecursiveCharacterTextSplitter to split the documents. However, I'm getting an error with @dqbd/tiktoken.

Here's the relevant code snippet:

  const textSplitter = new RecursiveCharacterTextSplitter({ chunkSize: 1000 });
  const newDocs = await textSplitter.splitDocuments(docs);
  const model = new OpenAI({
    temperature: 0,
  });
  const chain = loadSummarizationChain(model);

I've installed @dqbd/tiktoken to try to resolve the issue, but it hasn't worked. Here are the versions of the packages and Node.js that I'm using:

langchain: ^0.0.49 @dqbd/tiktoken: ^1.0.3 Node.js version: v18.X.X

Here's the error message that I'm seeing:

ERROR Invoke Error {
  "errorType": "Error",
  "errorMessage": "Please install @dqbd/tiktoken as a dependency with, e.g. `yarn add @dqbd/tiktoken`",
  "stack": [
    "Error: Please install @dqbd/tiktoken as a dependency with, e.g. `yarn add @dqbd/tiktoken`",
    "at importTiktoken (/var/task/index.js:747353:15)",
    "at process.processTicksAndRejections (node:internal/process/task_queues:95:5)",
    "at async OpenAIChat.getNumTokens (/var/task/index.js:762679:42)",
    "at async Promise.all (index 0)",
    "at async MapReduceDocumentsChain._call (/var/task/index.js:762999:26)",
    "at async MapReduceDocumentsChain.call (/var/task/index.js:746005:26)",
    "at async createSummaryDocOfShop (/var/task/index.js:873705:15)",
    "at async createIndexForShop (/var/task/index.js:873685:19)",
    "at async Runtime.handler (/var/task/index.js:877740:9)"
  ]
}

Additional Information:

I've tried uninstalling and reinstalling the @dqbd/tiktoken package, but it hasn't resolved the issue. I've also checked the documentation for langchain and @dqbd/tiktoken, but I haven't been able to find any solutions.