azure-openai-node
azure-openai-node copied to clipboard
Streaming tokens received out of order w/ Azure
When talking to my Azure deployment and using streaming mode, I've found that the tokens being returned are out of order. They're all there, just not in the right sequence. Is this a known issue? Here's my code, though it's unremarkable and largely follows the OAI cookbook:
const openaiExplainer = await openai.createChatCompletion({
model: OPENAI_MODEL_NAME,
messages: explainerArray,
temperature: 0,
stream: true
}, { responseType: 'stream' });
openaiExplainer.data.on('data', async (dataExplainer) => {
const linesExplainer = dataExplainer.toString().split('\n').filter(lineExplainer => lineExplainer.trim() !== '');
for (const lineExplainer of linesExplainer) {
const replyExplainer = lineExplainer.replace(/^data: /, '');
let parsedExplainer;
try {
parsedExplainer = JSON.parse(replyExplainer);
} catch (error) {
if (replyExplainer != '[DONE]') continue;
}
if (replyExplainer === '[DONE]' || (parsedExplainer && parsedExplainer.choices[0].finish_reason)) {
aiReplyChat.done = true;
return; // Stream finished
}
const deltaExplainer = parsedExplainer.choices[0].delta.content;
if (deltaExplainer) {
console.log(deltaExplainer)
aiReplyChat.explainer += deltaExplainer;
}
}
});
Sample output of the console from the above:
> Hello
> ,
> I
> Could
> you
> add
> it
> to
> my
> bill
> ?
>
>
> V
> ocabulary
>
>
> -
> mon
> (
> m
> ):
> my