rhubarb-lip-sync icon indicating copy to clipboard operation
rhubarb-lip-sync copied to clipboard

Support Opus audio files

Open pvanek opened this issue 10 months ago • 6 comments

https://opus-codec.org/

I can see following reasons to support it:

  • opus is quite easy to include (opensource, libopus)
  • it provides great results and quality in lower bit rates
  • some text to speech services (like e.g. Azure T2S) provide its output as mp3 or opus. It would allow to use Rhubarb out of the box without re-encoding in some pipelines

My only concerns are:

  • are you interested in this format for Rhubarb?
  • what are request to pass the review to include it in upstream?
    • is it preferred to copy the library into the src tree? Or to us system libs?
    • what are compilers/systems to support by a potential patch?
    • what is the test suite to pass?
  • what is the base branch to start developing? master or any of v2 ones?

pvanek avatar Feb 11 '25 07:02 pvanek

Support for Opus sounds great! I'd be happy to accept a MR.

  • Regarding the base branch: You can base it off master. The v2 branches are still experimental.
  • Regarding the libraries: In the master (v1) branch, I'm simply copying the repos to src/lib.
  • Regarding CI: See .github/workflows
  • Regarding tests: You can just extend the existing test suite.

DanielSWolf avatar Feb 12 '25 21:02 DanielSWolf

@DanielSWolf can you review a quick small patch? I'd like to have some pre-approval before the build/test/ci polishing. As you can see -- I basically stole the OggVorbisFileReader implementation. Of course it can/could benefit from some abstract class/inheritance, but I'd like to see the change smaller for now.

https://github.com/pvanek/rhubarb-lip-sync/commit/a6ab17fbdd73ca075c36484c1459a3382bca5a9f

pvanek avatar Feb 13 '25 14:02 pvanek

Thanks for the draft! Here's some quick feedback:

  • Right now, I agree that there would be little point in extracting common code. This may change with the next point, however:
  • Have you checked that your approach works on all platforms (especially Windows) if the file name contains Unicode characters? That's the reason for the wrapper functions read/seek/tellCallback. (I thought I had test cases for that, but then realized I'd only written them for the POC in Rust. At least for the new Opus code, I'd like to have similar tests, with something like filename-unicode-wide-😀🤣🙈🍨.opus.)
  • Make sure your code formatting matches the existing formatting. In particular, I'm indenting with tabs and placing opening braces at the end of lines.

The next step should be a PR, which will allow me to give more specific feedback.

DanielSWolf avatar Feb 14 '25 09:02 DanielSWolf

patch updated

  • code formatting fixed (brackets positioning, tabs)
  • utf8/emoji file names tested on Linux and Mac. Unfortunately I have no windows available at all. I hope I can find some of my colleagues next week.
  • I'm starting to provide snapshots of dependencies:
    • lib/opusfile-0.12/ -- super easy
    • lib/opus-1.5.2/ -- extremely complex with all configuration options avalable: https://github.com/xiph/opus/blob/main/CMakeLists.txt

pvanek avatar Feb 16 '25 16:02 pvanek

Please create a pull request.

DanielSWolf avatar Feb 17 '25 16:02 DanielSWolf

https://github.com/DanielSWolf/rhubarb-lip-sync/pull/164

pvanek avatar Feb 20 '25 14:02 pvanek