OpenAI-Unity icon indicating copy to clipboard operation
OpenAI-Unity copied to clipboard

feat: text to speech support (#103)

Open alphdevcode opened this issue 1 year ago • 2 comments

Summary

Text to speech feature was added to the OpenAI SDK.

Addresses #103

Code sample

This is the basic workflow for this feature:

[RequireComponent(typeof(AudioSource))]
public class TextToSpeech : MonoBehaviour
{
    private OpenAIApi openai = new OpenAIApi();
    
    void Start()
    {
        audioSource = GetComponent<AudioSource>();
  
        var request = new CreateTextToSpeechRequest
        {
            Input ="It feels so good to be able to talk!", // The text to be read aloud
            Model = "tts-1", // Text to speech model to use
            Voice = "alloy" // Voice to use
        };
        
        var response = await openai.CreateTextToSpeech(request);
        
        if(response.AudioClip) audioSource.PlayOneShot(response.AudioClip);
    }
}

Changes

  • Added the corresponding classes for handling the request creation to Runtime/DataTypes.cs.
  • Implemented the DispatchAudioRequest method to properly handle the response from the OpenAI API whether it's the generated audio or an error. None of the existing dispatch methods could be used due to the response not being in a json format, but a raw byte array.
  • Created the Runtime/Interfaces/IAudioResponse.cs following the same structure of the existing code for the dispatch methods.
  • Implemented the method CreateTextToSpeech. This is the method that will be called from outside the SDK.
  • Set up a sample scene to showcase the feature and add it to the package.json so it can be imported into a project like the other ones.

Checklist

  • [x] My code follows the style guidelines of this project.
  • [x] Self-review of my own code performed.
  • [x] No error nor warning in the console.
    • Note that OpenAI by default limits the text to speech models to 3 request per minute per session, if that limit is exceeded the request will return an error. This limit can be increased in the OpenAI dashboard: https://platform.openai.com/account/limits

alphdevcode avatar Jan 18 '24 02:01 alphdevcode

Any plans to add streaming capabilities to this?

BlakCake avatar Dec 06 '24 19:12 BlakCake

Hey @BlakCake I don't think the developer is willing to merge this PR. It's been here for almost a year

alphdevcode avatar Dec 09 '24 02:12 alphdevcode