flutter_tts icon indicating copy to clipboard operation
flutter_tts copied to clipboard

Added progress events to flutter_tts_web using onboundary

Open Robert01101101 opened this issue 1 year ago • 1 comments

Added progress events for Web by adding a listener to SpeechSynthesisUtterance's onboundary event. The fix / added feature is in order to add the missing support for progress events on the Web, I had reported an issue on that earlier today: Fixes #525

This is my first time contributing to an open-source project, I hope I adhered to the best practices 😅

Some notes on the changes:

  • The end index and current word are not provided by the onboundary event, so this solution uses a regex to iterate through the text, starting from the boundary start index, to find the next separator character. I'm not sure if the regex matches the logic used by the browser's TTS, but in my tests it did match.
  • The onboundary events might not exactly match other platform behaviors, for instance, there are separate events for new sentences, which this solution doesn't fire events for.
  • ttsState isn't being set, since it should be set by start, stop etc.

Robert01101101 avatar Sep 09 '24 03:09 Robert01101101

After running into some issues with this fix in my build, I want to add some additional notes. While my local debug build worked without issues, once I built for web, there were some limitations with the reliability of when onboundary events are fired.

From what I can tell:

  • Firing of onboundary events might be more reliable on some browsers (found no issues in Edge) than other browsers (Chrome had issues)
  • On affected browsers (Chrome), the issue seems to only appear when certain voices / languages are used. When the default language voice is used, it fires the events as expected, when certain other voices are used, it doesn't fire them at all.
  • Possibly related: https://stackoverflow.com/questions/51645956/speechsynthesisutterance-onboundary-event-not-firing-properly
    • One of the replies here mentions this suggestion for a fix: store the utterances in a variable with global context, but I'm not sure how to do that here, and if it would work. I tried assigning the utterance to a global variable, but that didn't seem to fix the limited support for onboundary on some browsers with some voices.

I'd be happy to do more investigation or work on this proposed change!

Robert01101101 avatar Sep 09 '24 22:09 Robert01101101

@Robert01101101 looks like there are some conflicts, please resolve when you have a moment so I can merge this in. Also any updates on your investigation?

dlutton avatar Nov 11 '24 16:11 dlutton

@dlutton I've resolved the merge conflict.

Regarding the investigation, I believe onboundary events are not supported at the moment for non-default voices.

Status: Won't Fix (Obsolete) The boundary event is not supported for all speech engines. Native speech synthesis on Mac OS X, Windows, and Chrome OS all support boundary events at the word level. Network-based speech synthesis, and Android speech synthesis don't currently support these. There's no immediate plans to fix those as the upstream APIs literally don't expose them at all. There's a feature request filed with Google's speech time to provide word timing information for the speech synthesis API that Chrome uses, but that's not available at this time.

For my use case, I updated my app to default to the native voice in the web build and disabled the option to switch voice. With that, it's working consistently.

My suggestion would be to include a note in the Readme. If you'd like I can take care of that, but I'm not quite sure how to add it to this PR. Can I commit the changes to my branch, push it, and that would update the changes included in this PR? I resolved the merge conflict using GitHub's conflict resolve UI and locally tested code with the merge conflict fix I copy pasted.

Robert01101101 avatar Nov 11 '24 23:11 Robert01101101

@Robert01101101 yes that would be great if you could update the README. You should be able to simply push to it.

dlutton avatar Nov 12 '24 18:11 dlutton

@dlutton I ended up adding 3 more small commits:

  • I noticed I forgot to adapt to the new interop types, I updated the code for consistency and tested it to verify it still works.
  • I updated the Readme to list Web support for speech marks, and explain limitations.
  • I noticed the Readme features section had repetition of top level bullet points: 'Android, iOS, macOS, Web, & Windows' and 'Android, iOS, Web, Windows & macOS'. I merged the two.

Let me know if I can do anything else or if there are any issues with this PR. And thanks for creating this package, I find it very useful!

Robert01101101 avatar Nov 12 '24 20:11 Robert01101101