w3c
Provide a way to specify custom encoder (to generate other types of formats)
Right now MediaRecorder can only encode in whatever format the browser can. This is at odds with building any kind of application with reproducible and expectable results - running the same code in different browsers might in some cases generate webm and in some other cases, mp4. Some users might be able to take the encoded media and use it immediately elsewhere but some others won't be able to do that. Non tech savvy users are going to be put off by this.
It would be great if instead of specifying the desired mimeType there was a way to provide the code for a custom encoder during MediaRecorder instantiation, and have the browser call that code periodically as it needs to encode more data.
How the code would look like or work like, I don't know. Perhaps a worker? A MediaRecorderWorker? A MediaRecorderEncoderWorker? But this would be in line with the Extensible Web Manifesto, in which browsers expose low-ish level APIs that let users build more advanced use cases on top. Right now the MediaRecorder API falls slightly short in that side.
This is the absolute number one feature that developers ask me about when I talk about MediaRecorder.
I think this is a very interesting use case that spreads beyond MediaRecoder and affects MediaStream (since it is the data provider to MediaRecorder). Say we have a MediaStream ---> Encoder ---> ContainerMultiplexer ---> (where MR would be the 'bag' containing the second and third steps).
@domenic and I have been discussing ways to reinterpret this chain:
- if MediaStream was a
ReadableStream(*), it would allow JS to create, e.g. aReadableStreamDefaultReader, and get easy access to the pixels of the MediaStream. This would simplify use cases that process the video feed to produce metadata, but it might not be viable to have a video encoder in JS in Real Time -- or would it? see e.g. whammy or lower complexity encoders such as animated GIF (e.g. this one ) - if MR could be instantiated with just an encoder(s) or a multiplexer, it would allow users to provide JS-defined multiplexers e.g. those supporting non-encoded formats (Y4M, FLAC), or to process the encoded data stream in JS.
In a nutshell, the use of Streams would make possible to 'tap' into the processing chain mentioned above, introducing custom elements instead.
(*) @domenic already investigated a version of the Media Recorder API for Streams in https://domenic.github.io/streaming-mediastreams/
Yes, most of the discussions I've had where we brainstormed how we would solve the issue of being able to encode into another format were about solution two: providing a JS defined encoder or multiplexer. Perhaps there could be some sort of prototype that custom code would need to implement in order to be a valid encoder/muxer. It could be required to implement functions or event listeners such as addNewFrame, or something of that sort.
On that same line too, yes, MediaStream, and more exactly MediaStreamTrack are 'a tad' opaque right now as well. We can only get tracks from sources, and it's somewhat hard to create tracks programmatically—we need to go through, say, a WebAudioContext to generate an audio track. Perhaps using Streams would help in this sense too—it could even enable client JS decoders?!
I would love to have this. For example, this would open up a whole world of potential encoder options on embedded platforms, where encoding is often implemented via hardware.
I feel like this could also solve https://github.com/w3c/mediacapture-record/issues/119 as it would allow people to write a custom encoder that buffers the video in memory and writes a seekable version at the end.
