Stream ConversationalRetrievalQAChain is not working

[mohamedcherifmo]

I'm using a pinecone vector with OpenAIChat model in a ConversationalRetrievalQAChain, it works as intended when im not streaming the output however if I change the OpenAIChat to streaming true as per the below code snippet it fails with the following error "TypeError: stream.getReader is not a function"

 const model = new OpenAIChat({
        streaming: true,
  callbackManager: CallbackManager.fromHandlers({
    async handleLLMNewToken(token: string) {
      console.log(token);
    },
  }),
    });

This is in a nextjs API route

[mohamedcherifmo]

I did find out that the reason for the issue is the axios-fetch-adapter is relying on a JS ReadableStream however NextJS readable stream is handled differently... The question now becomes is there a way when defining my call to override the fetch-adapter and create my own to cater for this and make it actually work with NextJS?

[Jasonkoolman]

@mohamedcherifmo Try using fetch-event-source

Example usage:

fetchEventSource('/api/chat', {
    method: 'POST',
    headers: {
        'Content-Type': 'application/json',
    },
    body: JSON.stringify({
        question,
        history,
    }),
    onmessage: (event) => {
        console.log(event)
    }
});

[mohamedcherifmo]

@Jasonkoolman where do u mean to try that?, the error is coming from the api router itself not the client side code

[eternnoir]

Same problem. But it's works file on 0.0.43

[KMKoushik]

It works fine on node v18 but getting the same error for node v16

[AlessandroAnnini]

I got 0.0.47, node v18 and fetch-event-source and it's not working for me

error:

TypeError: stream.getReader is not a function
  at getBytes (file:///app/node_modules/@fortaine/fetch-event-source/lib/esm/parse.js:2:27)
  at getResponse (file:///app/node_modules/langchain/dist/util/axios-fetch-adapter.js:206:19)
  at runMicrotasks (<anonymous>)
  at processTicksAndRejections (node:internal/process/task_queues:96:5)
  at async fetchAdapter (file:///app/node_modules/langchain/dist/util/axios-fetch-adapter.js:167:18) undefined

[nfcampos]

Hi, can you confirm if this isin a serverless or edge function?

[AlessandroAnnini]

I got it working using node 16 (not changing the dockerfile), using v0.0.47 and spinning a server on fly.io

it is wired because locally it was working with node 18 too

[BleedingDev]

I also have a problem with it in Next.js 13 + Node 19. :( Which detail can I provide? :)

[felipetodev]

same error with Next.js edge functions, [email protected] & [email protected]

[mrslimslim]

the problem still exists

[mrslimslim]

I think i find why it didnt work, it should be the misuse the origin web env "fetch" and "node-fetch". Check the detail here https://stackoverflow.com/questions/57664058/response-body-getreader-is-not-a-function.

Then I change the dep file event-source-parse getBytes function like this , problem solved.

export async function getBytes(stream, onChunk) {
    // const reader = stream.getReader();
    // let result;
    // // eslint-disable-next-line no-cond-assign
    // while (!(result = await reader.read()).done) {
    //     onChunk(result.value);
    // }
    stream.on('readable', () => {
        let chunk;
        while (null !== (chunk = stream.read())) {
            onChunk(chunk);
        }
    }
    );
}

[nfcampos]

@dqbd do you want to have a look at this one? Would be pretty strange if nextjs/vercel didn't support the ReadableStream web standard? Or is that people are running this in Node 16 without realising and Node 16 doesn't support ReadableStream

[nsoticek]

I think i find why it didnt work, it should be the misuse the origin web env "fetch" and "node-fetch". Check the detail here https://stackoverflow.com/questions/57664058/response-body-getreader-is-not-a-function.

Then I change the dep file event-source-parse getBytes function like this , problem solved.

export async function getBytes(stream, onChunk) {
    // const reader = stream.getReader();
    // let result;
    // // eslint-disable-next-line no-cond-assign
    // while (!(result = await reader.read()).done) {
    //     onChunk(result.value);
    // }
    stream.on('readable', () => {
        let chunk;
        while (null !== (chunk = stream.read())) {
            onChunk(chunk);
        }
    }
    );
}

This solves the issue. I had the same problem while trying to stream the response from the /experimental/autogpt function with node v17.9.1 in my local environment. getReader() function is not supported by Node.js node-fetch package.

[zhaoo]

This solution is fine! Can next version adapter browser fetch (node v18+) and node-fetch (node v16) ?