ai
ai copied to clipboard
Experimental speech streaming for LMNT (useChat/useCompletion React)
Summary
Adds speech streaming to useChat
and useCompletion
with streamData
.
-
useCompletion
&useChat
(for React) provide aexperimental_speechUrl
that can be used htmlaudio
elements - Integration functions for lmnt speech streams through
experimental_forwardLmntSpeechStream
-
streamData.experimental_appendSpeech
: add speech stream chunks to data stream (used automatically through forward functions) - Example:
examples/next-lmnt
: LMNT completion & chat speech streaming - Docs: LMNT provider docs, API docs for
experimental_forwardLmntSpeechStream
Notes
- The LMNT SDK does not work in the edge environment (as of
v1.1.2
)
This is exciting
This is awesome!
@lgrammel Hi Lars, I tried to test this one locally with no luck, it is showing this error:
⚠ ./app/api/chat-speech-elevenlabs/route.ts
Attempted import error: 'forwardModelFusionSpeechStream' is not exported from 'ai' (imported as 'forwardModelFusionSpeechStream').
I go to node_modules/ai and I see the function there, not sure if I need to do anything else. (I cloned the fork, checkout to the branch and run the example)
It is ready to test?
Thanks!
@lgrammel Hi Lars, I tried to test this one locally with no luck, it is showing this error:
⚠ ./app/api/chat-speech-elevenlabs/route.ts Attempted import error: 'forwardModelFusionSpeechStream' is not exported from 'ai' (imported as 'forwardModelFusionSpeechStream').
I go to node_modules/ai and I see the function there, not sure if I need to do anything else. (I cloned the fork, checkout to the branch and run the example)
It is ready to test?
Thanks!
Have you rebuilt the ai package? The easiest way is to just rebuild the whole repository (pnpm i
, pnpm build
) and then try out the example.
@lgrammel Hi Lars, I tried to test this one locally with no luck, it is showing this error:
⚠ ./app/api/chat-speech-elevenlabs/route.ts Attempted import error: 'forwardModelFusionSpeechStream' is not exported from 'ai' (imported as 'forwardModelFusionSpeechStream').
I go to node_modules/ai and I see the function there, not sure if I need to do anything else. (I cloned the fork, checkout to the branch and run the example) It is ready to test? Thanks!
Have you rebuilt the ai package? The easiest way is to just rebuild the whole repository (
pnpm i
,pnpm build
) and then try out the example.
That did the trick thank you!. I was doing npm run dev
, I did pnpm build
, npm start
and it worked.
It works really, really fast. I hope we can get this merged very soon.
Hi @MaxLeiter! did you have a chance to take a look?
Hello @lgrammel I saw that you changed from eleven labs to LMNT, there is a technical reason for this, eleven labs supports multi languages, LMNT still has no plans to launch this, wouldn't it be interesting to keep both options?
Thank you and congratulations for the excellent work
Hello @lgrammel I saw that you changed from eleven labs to LMNT, there is a technical reason for this, eleven labs supports multi languages, LMNT still has no plans to launch this, wouldn't it be interesting to keep both options?
Thank you and congratulations for the excellent work
Thanks. We want to use the official elevenlabs node SDK, but it does not support duplex streaming yet: https://github.com/elevenlabs/elevenlabs-js/issues/4
In the meantime, you could use modelfusion elevenlabs with the adapter that I had in an earlier version of this PR.
@lgrammel Hi! I can't find the example app for speech streaming in the Vercel AI SDK repo. where it's gone?
@lgrammel Hi! I can't find the example app for speech streaming in the Vercel AI SDK repo. where it's gone?
this feature has not been merged yet
Hi @MaxLeiter Can you merge this?
Hi @MaxLeiter Can you please approve this?
bump
we could really use this as well 🙏 thank you so much for the work on this
Any reason why this is closed? TTS is a great feature to have
Would be cool to see TTS added with the addition of gpt-4o
@lgrammel / @MaxLeiter Any follow up plans on adding TTS to vercel AI ?