ai
ai copied to clipboard
feat(isolate): add experimental audio isolation
Background
Continuing the migration of Orate into the Vercel AI SDK, I'm happy to introduce experimental_isolateAudio(), which can isolate speech (and eventually sound effects) from your audio!
Summary
This pull request introduces an experimental feature for audio isolation in the AI SDK. The changes include the addition of the isolateAudio function, updates to documentation, and new examples and tests to support the feature.
New Feature: Audio Isolation
.changeset/slimy-islands-warn.md: Added a patch for@ai-sdk/elevenlabs,@ai-sdk/provider, andaito introduce the experimental audio isolation feature.packages/ai/core/isolate-audio/index.ts: Exported theisolateAudiofunction andIsolationResulttype.packages/ai/core/isolate-audio/isolate-audio.ts: Implemented theisolateAudiofunction to isolate audio using an isolation model.packages/ai/core/isolate-audio/isolate-audio-result.ts: Defined theIsolationResultinterface to structure the result of anisolateAudiocall.packages/ai/core/isolate-audio/isolate-audio.test.ts: Added tests for theisolateAudiofunction to ensure proper functionality and error handling.
Documentation Updates
content/docs/03-ai-sdk-core/37-audio-isolation.mdx: Created a new documentation page for the audio isolation feature, including usage examples and settings.content/docs/07-reference/01-ai-sdk-core/13-isolate-audio.mdx: Added an API reference page for theisolateAudiofunction.content/docs/03-ai-sdk-core/index.mdx: Updated the index to include a link to the new audio isolation documentation.content/docs/07-reference/01-ai-sdk-core/index.mdx: Updated the reference index to includeisolateAudio().content/providers/01-ai-sdk-providers/90-elevenlabs.mdx: Added information about ElevenLabs isolation models and their capabilities.
Examples
examples/ai-core/src/isolate-audio/elevenlabs.ts: Added an example script demonstrating how to use theisolateAudiofunction with ElevenLabs.
Codebase Updates
packages/ai/core/index.ts: Exported theisolateAudiofunction from the core package.
Tasks
- [x] Tests for the changes have been added (for bug fixes / features)
- [x] Docs have been added / updated (for bug fixes / features)
- [x] If required, a patch changeset for relevant packages has been added
- [x] You've run
pnpm prettier-fixto fix any formatting issues
Future Work
New PRs will be opened for providers.
Please develop against v5 branch. I'll review once this PR is targeting v5.
Sorry, we didn't get to this in time. Please re-create the PR if the change is still needed and we'll take a look! This comment is automated as part of our triage process. Truly sorry for the inconvenieince, we are doing our best 🖤