azure-openai-node icon indicating copy to clipboard operation
azure-openai-node copied to clipboard

Streaming tokens received out of order w/ Azure

Open joshforman opened this issue 1 year ago • 1 comments

When talking to my Azure deployment and using streaming mode, I've found that the tokens being returned are out of order. They're all there, just not in the right sequence. Is this a known issue? Here's my code, though it's unremarkable and largely follows the OAI cookbook:

const openaiExplainer = await openai.createChatCompletion({
            model: OPENAI_MODEL_NAME,
            messages: explainerArray,
            temperature: 0,
            stream: true
          }, { responseType: 'stream' });

          openaiExplainer.data.on('data', async (dataExplainer) => {
            const linesExplainer = dataExplainer.toString().split('\n').filter(lineExplainer => lineExplainer.trim() !== '');
            for (const lineExplainer of linesExplainer) {
              const replyExplainer = lineExplainer.replace(/^data: /, '');

              let parsedExplainer;
              try {
                parsedExplainer = JSON.parse(replyExplainer);
              } catch (error) {
                if (replyExplainer != '[DONE]') continue;
              }

              if (replyExplainer === '[DONE]' || (parsedExplainer && parsedExplainer.choices[0].finish_reason)) {
                aiReplyChat.done = true;
                return; // Stream finished
              }

              const deltaExplainer = parsedExplainer.choices[0].delta.content;

              if (deltaExplainer) {
                console.log(deltaExplainer)
                aiReplyChat.explainer += deltaExplainer;
              }
            }
          });

Sample output of the console from the above:

>  Hello
>  ,
>   I
>   Could
>   you
>   add
>   it
>   to
>   my
>   bill
>  ?
>  
>  
>  V
>  ocabulary
>  
>  
>  -
>   mon
>   (
>  m
>  ):
>   my

joshforman avatar May 23 '23 17:05 joshforman