ai icon indicating copy to clipboard operation
ai copied to clipboard

feat(isolate): add experimental audio isolation

Open haydenbleasel opened this issue 7 months ago • 1 comments

Background

Continuing the migration of Orate into the Vercel AI SDK, I'm happy to introduce experimental_isolateAudio(), which can isolate speech (and eventually sound effects) from your audio!

Summary

This pull request introduces an experimental feature for audio isolation in the AI SDK. The changes include the addition of the isolateAudio function, updates to documentation, and new examples and tests to support the feature.

New Feature: Audio Isolation

  • .changeset/slimy-islands-warn.md: Added a patch for @ai-sdk/elevenlabs, @ai-sdk/provider, and ai to introduce the experimental audio isolation feature.
  • packages/ai/core/isolate-audio/index.ts: Exported the isolateAudio function and IsolationResult type.
  • packages/ai/core/isolate-audio/isolate-audio.ts: Implemented the isolateAudio function to isolate audio using an isolation model.
  • packages/ai/core/isolate-audio/isolate-audio-result.ts: Defined the IsolationResult interface to structure the result of an isolateAudio call.
  • packages/ai/core/isolate-audio/isolate-audio.test.ts: Added tests for the isolateAudio function to ensure proper functionality and error handling.

Documentation Updates

  • content/docs/03-ai-sdk-core/37-audio-isolation.mdx: Created a new documentation page for the audio isolation feature, including usage examples and settings.
  • content/docs/07-reference/01-ai-sdk-core/13-isolate-audio.mdx: Added an API reference page for the isolateAudio function.
  • content/docs/03-ai-sdk-core/index.mdx: Updated the index to include a link to the new audio isolation documentation.
  • content/docs/07-reference/01-ai-sdk-core/index.mdx: Updated the reference index to include isolateAudio().
  • content/providers/01-ai-sdk-providers/90-elevenlabs.mdx: Added information about ElevenLabs isolation models and their capabilities.

Examples

  • examples/ai-core/src/isolate-audio/elevenlabs.ts: Added an example script demonstrating how to use the isolateAudio function with ElevenLabs.

Codebase Updates

  • packages/ai/core/index.ts: Exported the isolateAudio function from the core package.

Tasks

  • [x] Tests for the changes have been added (for bug fixes / features)
  • [x] Docs have been added / updated (for bug fixes / features)
  • [x] If required, a patch changeset for relevant packages has been added
  • [x] You've run pnpm prettier-fix to fix any formatting issues

Future Work

New PRs will be opened for providers.

haydenbleasel avatar Apr 12 '25 18:04 haydenbleasel

Please develop against v5 branch. I'll review once this PR is targeting v5.

lgrammel avatar Apr 13 '25 07:04 lgrammel

Sorry, we didn't get to this in time. Please re-create the PR if the change is still needed and we'll take a look! This comment is automated as part of our triage process. Truly sorry for the inconvenieince, we are doing our best 🖤

gr2m avatar Nov 07 '25 17:11 gr2m