HLS Support for OffthreadVideo

[sydneypalumbo]

Hi there! I'm currently trying to use m3u8 files (from mux.com) as the src of <OffthreadVideo /> with remotion v4.0.19 but getting the following error during rendering. I thought this would be supported with ffmpeg, but is it not possible to use m3u8 with OffthreadVideo? Here's an example file, which doesn't seem to be corrupt. Thanks so much in advance for your help!

>  [compositor] [FFmpeg] Statistics: 2196 bytes read, 0 seeks
>  Error occurred Error: Compositor error: Invalid data found when processing input
>     0: std::backtrace::Backtrace::force_capture 
>     1: <compositor::errors::ErrorWithBacktrace as core::convert::From<ffmpeg_next::util::error::Error>>::from
>     2: compositor::opened_stream::open_stream
>     3: compositor::opened_video_manager::OpenedVideoManager::get_video
>     4: compositor::commands::execute_command
>     5: rayon_core::thread_pool::ThreadPool::install::{{closure}}
>     6: <rayon_core::job::StackJob<L,F,R> as rayon_core::job::Job>::execute
>     7: rayon_core::registry::WorkerThread::wait_until_cold
>     8: std::sys_common::backtrace::__rust_begin_short_backtrace
>     9: core::ops::function::FnOnce::call_once{{vtable.shim}}
>    10: std::sys::unix::thread::Thread::new::thread_start
>    11: __pthread_joiner_wake
>      at onMessage (/.../node_modules/@remotion/renderer/dist/compositor/compositor.js:56:47)
>      at processInput (/.../node_modules/@remotion/renderer/dist/compositor/compositor.js:122:9)
>      at Socket.<anonymous> (/.../node_modules/@remotion/renderer/dist/compositor/compositor.js:142:9)
>      at Socket.emit (node:events:514:28)
>      at Socket.emit (node:domain:489:12)
>      at addChunk (node:internal/streams/readable:324:12)
>      at readableAddChunk (node:internal/streams/readable:297:9)
>      at Socket.Readable.push (node:internal/streams/readable:234:10)
>      at Pipe.onStreamRead (node:internal/stream_base_commons:190:23)

[JonnyBurger]

Thanks a lot for reporting!

Actually, there is no explicit documentation on which codecs are supported by OffthreadVideo. I am fixing that with #2934.

We'll also add a FFmpeg binary which supports HLS in the next version after which we can assess whether this file can be supported.

[JonnyBurger]

The new version now throws a different error and it seems like it will take significant effort to support M3U8.

Until we sort it out, I recommend to enable MP4 files on Mux and use those to render!

[sydneypalumbo]

@JonnyBurger sounds good, thanks again!

[mjdavidson]

@JonnyBurger we're also very keen to use HLS with <OffthreadVideo />, to preview video edits on our front-end. How far away do you think support for this will be?

[JonnyBurger]

@mjdavidson If you just need it in the preview / Remotion Player, you can probably add hls.js to get it working!

For rendering, we need to parse the HLS format and download only the specific parts before feeding it to OffthreadVideo.

The scariest part is the HLS spec which seems quite long. I don't see a suitable library for JS/Rust that we can get working quickly, so I don't see it happening soon.

[mjdavidson]

@JonnyBurger yep, when we render the final video we have access to a normalized mp4 video so we will use that. We already use hls.js in our React project for our normal video player. How do you propose linking hls.js to the Remotion video component?

[JonnyBurger]

@mjdavidson Sure, I added a snippet for it here: https://www.remotion.dev/docs/miscellaneous/snippets/hls

Only tested some very basic cases though.

[rmanalan]

I got this working pretty well today. I did end up adding a few optimizations to the useEffect in the guide:

  useEffect(() => {
    if (!src) {
      throw new Error('src is required')
    }

    const hls = new Hls({
      enableWorker: !isSafari(),
      lowLatencyMode: true,
      backBufferLength: 20,
      startLevel: 4,
      maxBufferLength: 5,
      maxMaxBufferLength: 5,
    })
    if (hls) {
      hls.on(Hls.Events.MANIFEST_PARSED, () => {
        hls.startLoad(startFrom)
      })
      hls.attachMedia(videoRef.current!)
      hls.loadSource(src)
    }

    return () => {
      hls.destroy()
    }
  }, [src])

Also, the guide states that audio "will not render," but in my testing, audio works just fine in preview mode.

[JonnyBurger]

@rmanalan Thanks for sharing! Putting this back into the docs: https://github.com/remotion-dev/remotion/pull/3155

[JonnyBurger]

By "will not render", I meant actually turning the video into an MP4, which I think is true. I'll clarify it as well.

[tslater]

@JonnyBurger Can you give a little more detail as to what errors you're getting or what you need to feed to the component? I'd like to try my hand at getting hls to work if you can give me the context.

[JonnyBurger]

@tslater I don't know the errors I got specifically, I just know that the native <video> tag doesn't support HLS by default and the FFmpeg C library doesn't recognize a .m3u8 as media like other video files.

It is just unknown to them which is why it doesn't work right now.

[algora-pbc[bot]]

💎 $226.11 bounty • nonsense.com

Steps to solve:

1. Start working: Comment /attempt #2930 with your implementation plan
2. Submit work: Create a pull request including /claim #2930 in the PR body to claim the bounty
3. Receive payment: 100% of the bounty is received 2-5 days post-reward. Make sure you are eligible for payouts

Thank you for contributing to remotion-dev/remotion!

Add a bounty • Share on socials

Attempt	Started (GMT+0)	Solution
🟢 @feliciien	Mar 25, 2024, 2:24:20 PM	WIP
🟢 @ChetanBhosale	Jun 30, 2024, 11:51:20 AM	WIP

[urbit-pilled]

Looks like something I can fix, I'll give it a try /attempt #2930

[JonnyBurger]

I think this is a really difficult issue to solve (at least for me 😄)

Do you have a plan on how to proceed? @urbit-pilled

(The bounty was not created by us nor someone we know!)

[feliciien]

/attempt #2930

Algora profile	Completed bounties	Tech	Active attempts	Options
@feliciien	5 bounties from 1 project	MDX, Rust, JavaScript & more		Cancel attempt

[feliciien]

@sydneypalumbo Hello can you tell me what command? Did you run for having this "> [compositor] [FFmpeg] Statistics: 2196 bytes read, 0 seeks

Error occurred Error: Compositor error: Invalid data found when processing input 0: std::backtrace::Backtrace::force_capture 1: <compositor::errors::ErrorWithBacktrace as core::convert::From<ffmpeg_next::util::error::Error>>:: from 2: compositor::opened_stream::open_stream 3: compositor::opened_video_manager::OpenedVideoManager::get_video 4: compositor::commands::execute_command 5: rayon_core::thread_pool::ThreadPool::install::{{closure}} 6: <rayon_core::job::StackJob<L,F,R> as rayon_core::job::Job>::execute 7: rayon_core::registry::WorkerThread::wait_until_cold 8: std::sys_common::backtrace::__rust_begin_short_backtrace 9: core::ops::function::FnOnce::call_once{{vtable.shim}} 10: std::sys::unix::thread::Thread::new::thread_start 11: __pthread_joiner_wake at onMessage (/.../node_modules/@remotion/renderer/dist/compositor/compositor.js:56:47) at processInput (/.../node_modules/@remotion/renderer/dist/compositor/compositor.js:122:9) at Socket. (/.../node_modules/@remotion/renderer/dist/compositor/compositor.js:142:9) at Socket.emit (node:events:514:28) at Socket.emit (node:domain:489:12) at addChunk (node:internal/streams/readable:324:12) at readableAddChunk (node:internal/streams/readable:297:9) at Socket.Readable.push (node:internal/streams/readable:234:10) at Pipe.onStreamRead (node:internal/stream_base_commons:190:23)" and where I want to reproduce the same error so it will be more easy for me to fix it

[feliciien]

I want to reproduce the same error on my editor

[g3Bg2]

@JonnyBurger

Is there a time limit to a video?

There should be a time limit to the video because we can play live stream via the .m3u8 link.

I tried playing a video, it gets the packets(segments) in the network tab but the video player only plays the video once it gets all the packets(segments).

[ChetanBhosale]

/attempt #2930

Options

• Cancel my attempt

[JonnyBurger]

We have not put out this bounty and don't have a clear path towards solving this issue. This is very hard, don't attempt it unless you know exactly what to do!

[Asmitpadia]

Instead of usign OffthreadVideo we can use a library for react video.js and can also inspect that the m3u8 file is correct or not. If you are interested we can try this