mediacapture-record
mediacapture-record copied to clipboard
w3c
Input width, height MUST be recorded and playable
Input MediaStreamTrack of kind "video" width and height MUST be recorded and playable.
It seems reasonable to expect the video to be recorded with dynamically changing resolution, so that input frames are correctly represented in the output. The main issue I see though, is if recording into a container that doesn't support changing resolution dynamically. Should that be an error? Or should it be left to the UA to take an alternative approach, like scale, crop, etc?
I don't know the container landscape well enough to say whether, or which, containers support this, but I imagine there are some that don't. Others might know better.
It seems reasonable to expect the video to be recorded with dynamically changing resolution, so that input frames are correctly represented in the output. The main issue I see though, is if recording into a container that doesn't support changing resolution dynamically. Should that be an error? Or should it be left to the UA to take an alternative approach, like scale, crop, etc?
I don't know the container landscape well enough to say whether, or which, containers support this, but I imagine there are some that don't. Others might know better.
AFAIK the only container that Firefox and Chromium support is Matroska (multiple codecs supported), or Webm (Matroska with VP8, VP9, was Vorbis, now Opus). The codecs that those browsers' implement contains code which checks the width and height of input.
This is the edge case (observable at Chromium) which led to filing this issue https://bugs.chromium.org/p/chromium/issues/detail?id=972470 and this PR. Perhaps there is a bug in the code. Can you get the recording the output the width and height changes at Chromium?
<!DOCTYPE html>
<html>
<head>
</head>
<body>
<video id="sink" autoplay muted controls></video>
<script>
(async() => {
const sink = document.querySelector("video");
const canvas = document.createElement("canvas");
const imageData = [{
width: 50,
height: 50,
fillStyle: "green"
}, {
width: 150,
height: 150,
fillStyle: "yellow"
}, {
width: 300,
height: 300,
fillStyle: "red"
}];
const segments = 60;
const videoData = [];
const ctx = canvas.getContext("2d");
const canvasStream = canvas.captureStream(0);
const [videoTrack] = canvasStream.getVideoTracks();
const mediaStream = [canvasStream, videoTrack].find(({requestFrame:rF}) => rF);
const recorder = new MediaRecorder(canvasStream);
const result = new Promise(resolve =>
recorder.ondataavailable = e => {
sink.load();
sink.srcObject = null;
resolve(sink.src = URL.createObjectURL(e.data));
});
recorder.start();
videoTrack.onunmute = e => console.log(e);
videoTrack.onmute = e => console.log(e);
videoTrack.onended = e => console.log(e);
sink.onloadedmetadata = e => console.log(e.type, e.target.tagName, sink.videoWidth, sink.videoHeight);
sink.onresize = e => console.log(e.type, e.target.tagName, sink.videoWidth, sink.videoHeight);
sink.srcObject = canvasStream;
for (const {width, height, fillStyle} of imageData) {
// At Chromium 76 the only code that have been able to set width and height with
// results in the incorrect value for height: 150 when width: 300
// for last 60 frames, should be width: 300 when height: 300
// ({width:canvas, height:canvas});
// Object.assign(canvas, {width, height});
console.log(canvas.width, canvas.height, {width, height});
// at Firefox 69 each set width and heiht of canvas
// is reflected at resulting webm video by settin width and height
// this should be the only setting needed and output the same result at Chromium
canvas.width = width;
canvas.height = height;
await new Promise(async resolve => {
for (let i = 0; i < segments; i++) {
await new Promise(promise =>
requestAnimationFrame(function draw() {
ctx.clearRect(0, 0, width, height);
ctx.fillStyle = fillStyle;
ctx.fillRect(0, 0, width, height);
mediaStream.requestFrame();
promise();
})
)
}
resolve()
})
}
recorder.stop();
videoTrack.stop();
videoTrack.enabled = false;
console.log(await result, sink.srcObject, canvasStream, videoTrack);
})();
</script>
</body>
</html>
Interestingly, while experiementing with code other than the edge at Chromium the width and height is recorded when <canvas> width and height is changed.
If a container does not support variable width the container should not be implemented by MediaRecorder.
Had not considered whether or not an error should be thrown. At first impression, variable width and height input should "just work", else the entire concept of canvas.captureStream() needs to be revisted and constraints should be deleted from the several revelant specifications which state they could remotely have an effect on a MediaStreamTrack. The error would be with the implementation by browser authors. Though if your question is essentially if an error should not be thrown then what is the point of this PR then yes, throw the error; which would require some internal code and testing to verify that the output width and height corresponds to the input width and height - which should already be implemented.
@Pehrsons Note, Mozilla appears to be reluctant to add support for container other than WebM and codecs other than VP8 and Opus https://bugzilla.mozilla.org/show_bug.cgi?id=1422891#c11, https://hg.mozilla.org/mozilla-central/annotate/58f1d879059c62ddeaf4b4fcf2624fb2c66d6ab9/dom/media/MediaRecorder.cpp#l1488
@Pehrsons Note, Mozilla appears to be reluctant to add support for container other than WebM and codecs other than VP8 and Opus https://bugzilla.mozilla.org/show_bug.cgi?id=1422891#c11, https://hg.mozilla.org/mozilla-central/annotate/58f1d879059c62ddeaf4b4fcf2624fb2c66d6ab9/dom/media/MediaRecorder.cpp#l1488
I think you're overanalyzing things.
@Pehrsons The Status RESOLVED WONTFIX at Webm support for h.264 codec https://bugzilla.mozilla.org/show_bug.cgi?id=1562862 is a clear indication that Mozilla appears to be reluctant to add support for container other than WebM and codecs other than VP8 and Opus (save for Ogg). If the codecs for MediaRecorder are or were intended to be fixed, static, then VP9 should not be supported either. *oogle Chrome and Chromium currently DO support video/webm;codecs=h264, video/webm;codecs=avc1, video/x-matroska;codecs=h264 and video/x-matroska;codecs=avc1, in spite of any purported limitation of the codecs that WebM was intended to support.
@Pehrsons For clarity https://groups.google.com/a/webmproject.org/forum/#!topic/webm-discuss/nJ39yu-CuKI ; https://bugzilla.mozilla.org/show_bug.cgi?id=1563140
It seems reasonable to expect the video to be recorded with dynamically changing resolution, so that input frames are correctly represented in the output. The main issue I see though, is if recording into a container that doesn't support changing resolution dynamically. Should that be an error? Or should it be left to the UA to take an alternative approach, like scale, crop, etc?
I don't know the container landscape well enough to say whether, or which, containers support this, but I imagine there are some that don't. Others might know better.
@Pehrsons Interestingly see Unexpected Error when appending webm files: mkvmerge throws error when width and height are different at https://gitlab.com/mbunkus/mkvtoolnix/issues/2582#note_189360046
This is not a bug. Matroska doesn't support changing pixel widths/heights in the middle of the stream.
Though that is evidently not the case as demonstrated by the output of MediaRecorder at Firefox. If that were/is the case per some inconspicuous note or detail at a controlling Matroska or WebM specification then Firefox is beyond the cutting edge as to writing Matroska-based WebM files and further the Firefox implementation needs to be the example of how such functionality IS achieved, which should lead to the relevant specifications being updated to reflect what is actually possible using the Matroska-based webm container.
@Pehrsons Interestingly see Unexpected Error when appending webm files: mkvmerge throws error when width and height are different at https://gitlab.com/mbunkus/mkvtoolnix/issues/2582#note_189360046
This is not a bug. Matroska doesn't support changing pixel widths/heights in the middle of the stream.
I haven't found any evidence this is true. There's certainly nothing in the standard (https://www.matroska.org/technical/specs/index.html) that explicitly precludes this that I could find. I'd also point out if you read Matroska's goals as a project (https://www.matroska.org/technical/whatis/index.html), I would fully expect them to support this. Format changes mid-stream are possible (and even common) in many video codecs, including major ones like H.264 and VP8/VP9.
It is true that Matroska has a container level width/height that gets set, and so it's possible that value could be misleading if a client is expecting no change to the format of the stream--but I also don't think it's unreasonable to set those values to whatever the video stream starts at, and expect clients to be able to understand stream-level format changes that roll through as they process data from the container.
Also ffmpeg handles dynamic resolution changes just fine in mkv (I've been processing mkv's with dynamic format changes via ffmpeg for years now).
And one more comment: I'd argue mkvmerge does not support dynamic format changes, because they'd need to construct stream-level data to indicate a format change (completely possible without re-encoding, btw). But that's a very different thing from mkv not handling dynamic format changes mid-stream.
@jnoring Re-opened the issue as could also not locate any specific language in the relevant specifications where Matroska container and derived WebM format restricts variable input stream width and height. Re-opened the issue; the author of mkvmerge acknowledges the above facts https://gitlab.com/mbunkus/mkvtoolnix/issues/2582#note_189460892
You're right that the specs don't forbid it. However, this only works for video formats that have their width & height encoded at the bitstream level for each key frame as there's only one place those values can be stored at the container level. This isn't the case for all video codecs. I can remove this check for codecs that do contain that information, which include VP8, VP9, AV1, h.264 & h.265 (and with that the most popular codecs today).
@jnoring mkvmerge now supports variable width and height in same container (mkvmerge: allow appending video tracks with differing pixel dimensions https://gitlab.com/mbunkus/mkvtoolnix/commit/2548a7f3497940135e55be604a87b9495fdff94d).
However, there are at least two Chromium bugs that prevent that browser from rendering variable width and height of input video track. One bug is linked at https://github.com/w3c/mediacapture-record/pull/172#issuecomment-506397242. Even when not using MediaRecorder, Chromium browser implementation of HTMLMediaElement does not dispatch resize event when video track pixel dimensions change within a webm file nor display the correct videoWidth and videoHeight of input file at <video> element https://bugs.chromium.org/p/chromium/issues/detail?id=983777. Until those bugs are fixed, even if this PR was included in the W3C MediaRecorder specification the current HTMLVideoElement implementation at Chromium will not display the resulting webm file variable width and height correctly at <video> within HTML (Issue 983777: HTMLVideoElement does not resize or display correct videoWidth or videoHeight https://bugs.chromium.org/p/chromium/issues/detail?id=983777).
Closing pull request due to IPR issue (guest e pi is not allowed to make substantive contributions). Please continue discussion in bug #173.
