Add stdin stream support.

[PaulKinlan]

Many applications stream STDIN to allow ffmpeg to encode from a data source that is not a file, or a data source that will never end (i.e, livestreaming a web cam).

Right now the lib can't support it because stdin read in WASM needs to be blocking (which freezes the thread) and this means the event loop is frozen too so that data that would have been sent via postMessage can't be processed.

Describe the solution you'd like I added stdin buffer support to a fork of the other ffmpeg.js that allows STDIN to be read inside a worker. https://github.com/PaulKinlan/ffmpeg.js/pull/1

It uses SharedArrayBuffers to communicate between the window and the worker, so this limit's it to Chrome on Desktop at the moment. It does however allow ffmpeg to block on stdin and then recieve data from the window asynchronously.

[jeromewu]

This is an interested one, you can check the latest version (^0.8.1) to see if we can do it after the major refectoring. I will take a look when available, thanks.

[PaulKinlan]

Yep. i can certainly have a look.

[ROBERT-MCDOWELL]

Just following up about this important feature. Is the last git can let us experiment it?

[deepesh-agarwal]

This feature would be very helpful.

[Banou26]

Is there an ETA on that major refactoring ?

after the major refectoring

If the refactor is not gonna happen(since i have not seen any commits on this repo & your fork of FFMPEG for months), I'm really interested in the stdin stream support and would like to work on it if possible, but i have very small knowledge of any native/wasm tools that this project uses, so if you could tell me where to start/what to look for i'd really appreciate it.

[creativefctr]

@jeromewu any updates on this?

[jeromewu]

@creativefctr sadly, the answer is no and I would say it is hard to achieve considering the limitation we have right now.

[bradisbell]

Hello @jeromewu. The limitation you mentioned... is that lack of SharedArrayBuffer support in browsers? It seems that SharedArrayBuffer is available again. Is it now possible to add STDIN support? Or, is there some other limitation?

[jeromewu]

Hmm, it is not SharedArrayBuffer that introduces this limitation. It is more like the current state of WebAssembly cannot handle stdin, so I cannot find a way to support stdin at the moment.

[Banou26]

Not sure if most people wanna get into this but you can pretty easily make a small LibAV C++ API compiled to WASM that deals with only computing when necessary to stream data in and out of WASM-land, this is obviously not as simple as this library's plug and play API but if you really want it you can manage to do smth with a little bit of work, I have almost no prior experience with C++ but still managed to make a transmuxer work with streaming in WASM.

[jeromewu]

Hi @Banou26, that sounds cool! Would you mind sharing a working sample code that handles streaming? Thanks.

[Banou26]

Hey, sorry, sadly i cannot share my actual code as it's very project-specific, If/When i have more time i potentially wanna revisit it to make it an actual library to share with everyone.

Though what i can do now is give you a rough explanation on how it works.

You can basically retake this example https://github.com/leandromoreira/ffmpeg-libav-tutorial/blob/master/2_remuxing.c

And modify it in a way that it doesn't use the filesystem but in memory buffers pushed from javascript.

But most importantly, you need to replace the main loop, while(true) by something non blocking, so for example, you could make it process 5GOPs per javascript calls. This is to leave the thread it's currently running on, time for javascript to interact with it, e.g push more buffers, get back the previously processed buffers, make more process calls, ect...

So in the end the API you end up with is something like this in javascript

const remuxer = (libavInstance ?? new libavInstance = await (await import('../dist/libav.js'))()).Remuxer(size)
remuxer.init(BUFFER_SIZE)

const process = () => {
  remuxer.clearInput()
  remuxer.push(buffer)
  remuxer.process()
  transmuxedBuffer.push(new Uint8Array(remuxer.getInt8Array()))
  remuxer.clearOutput()
}

process()

// waits for more data to download

process()

// ...ect

remuxer.close()

Which you can then conveniently turn into a Stream based API for ease of use.

Only problem with your project i can see, is that the entire codebase of FFMPEG is based around the fact that it expects to be run on a FS with the full video at its disposition, you could probably hack it into a way that it uses the remote fetching mechanism because this code probably has some IO waiting mechanism, but it still would require probably a LOT of hacking to make it work because of the nature of this project being based on the actual FFMPEG CLI codebase.

Hope this gave you ideas/a start on how you could potentially implement it for this project. If you have any more questions i'd be happy to try to answer them as much as possible.

[jeromewu]

@Banou26 Thanks for the explanation, it helps a lot. Right now looks like FS is the bottleneck for the whole thing to happen, I will think about it and see if there is a way to keep most of the features while not to use FS.

[Banou26]

@davedoesdev managed to asyncify ffmpeg.js's emscripten_read_async to get streaming outputs https://github.com/Kagami/ffmpeg.js/pull/166 so we could potentially manage to asyncify emscripten's stdin/stdout's to finally get streaming support?

[vKongv]

I am planning to work on a live stream studio web app that allow our streamer to customise their stream with on the web app before they stream to Facebook. MVP require a few key features:

• Stream from Webcam
• Work in both web and mobile
• Add on screen banner that showcase product information
• Able to stream to Facebook

I am looking for ways to actually do it and is currently experimenting ffmpeg.wasm which bump into the problem of reading constant stream from webcam to ffmpeg before modifying it. Most of the solution online is recording a video and post process it but there weren't any reliable source for live streaming editing.

If anyone could shine me some light on how to achieve this would be greatly appreciated.

[Bessonov]

I have a similar use case as @HamptonMakes in https://github.com/ffmpegwasm/ffmpeg.wasm/issues/141 , but to convert webm to mp4 container and subsequently transport it over WebSocket.

It seems like MEMFS could be used to simulate stdin/stdout, as shown in https://github.com/Kagami/ffmpeg.js#user-content-files .

@HamptonMakes @vKongv have you found a solution?

[Banou26]

If anyone ever wants a simple streaming remuxing lib for the web, i've just opened https://github.com/Banou26/libav-wasm (for now it simply remux mkv->mp4 with seeking and streaming support) I'll try to clean it up a little bit, publish it on npm & add features if anyone actually wants it.