Add support for urn:3gpp:video-orientation RTP header extension.

[steely-glint]

We have a camera WHIP video source that is occasionally mounted upside down. It correctly sets the urn:3gpp:video-orientation header extension. Unfortunately Galene doesn't support it. I think all we need is pass-through of the extension - no actual video wrangling - the receiving browser can do all the heavy lifting.

[jech]

I think it needs negotiating, so we cannot just blindly pass it through. See 3GPP TS 26.114 Section 6.2.3.3.

Could you please check whether the camera includes the extension in the a=extmap attribute of the SDP?

[steely-glint]

It does include the extmap , I even wrote a test page in Chrome. (I can clean it up and publish it next week if that would be useful).

Chrome's behaviour is bizarre - it negotiates fine with a peer that doesn't support the extension (e.g. Galene) - but then only sends packets that don't have the extension, not those that do.

[jech]

Chrome's behaviour is bizarre - it negotiates fine with a peer that doesn't support the extension (e.g. Galene) - but then only sends packets that don't have the extension, not those that do.

Do you mean that Chrome drops the whole packet rather than just dropping the extension?

I guess we should negotiate the extension, both on the receiving and the sending side, and then pass through the extension even if the receiver didn't negotiate it. If anything goes wrong, we can revisit the decision.

[steely-glint]

Yep, as far as I can tell (from webrtc-internals) chrome simply doesn't send the packets with extensions unless the answerer supports it. - when you send (on the same transceiver) a frame without the rotation it goes through fine.

(I also mentioned this in the pion discord, so there may be some movement there too)

[jech]

Could you please describe how to trigger usage of the extension in Chrome?

[steely-glint]

Using insertable streams is the easiest way:

    function rotateTrack(vt, a) {

        let ret = vt;
        try {
            const processor = new MediaStreamTrackProcessor({track: vt.clone()});
            const generator = new MediaStreamTrackGenerator({kind: 'video'});
            let videoWorker = new Worker("rotationWorker.js");
            let readable = processor.readable;
            let writeable = generator.writable;
            videoWorker.postMessage({'readable': readable, 'writeable': writeable, angle: a}, [readable, writeable]);
            ret = generator;
        } catch (e) {
            console.log(e);
        }
        return ret;
    }

    function setupVideo() {
        var gumConstraints = {video: true};
        var promise = new Promise(function (resolve, reject) {
            navigator.mediaDevices.getUserMedia(gumConstraints)
                .then((stream) => {
                    console.log("add local stream");
                    let vt = stream.getVideoTracks();
                    if (vt.length == 1) {
                        let track = vt[0];
                        track = rotateTrack(track, 180);
                        let me = document.getElementById("local");
                        me.srcObject = new MediaStream([track]);
                        gumPc.addTrack(track);
                    }
                    resolve(false);
                })
                .catch((e) => {
                    console.log('getUserMedia() error:' + e);
                    reject(e);
                });
        });
        return promise;
    }

rotationWorker.js

onmessage = async (me) => {
    let readable = me.data.readable ;
    let writable = me.data.writeable;
    let angle = me.data.angle;


    const t3 = new TransformStream({
        async transform(videoFrame, controller) {

            let newFrame = new VideoFrame(videoFrame, {
                'rotation': angle,
            });
            videoFrame.close();
            controller.enqueue(newFrame);
        }
    });
    await readable.pipeThrough(t3).pipeTo(writable);

}

I'll try and clean up my test code and publish it tomorrow - it is currently littered with false starts and secret WHIP tokens :-(

[steely-glint]

Here's my test page - you'll want to change the WHIP/ viewer URLS for local testing.

https://dev.pi.pe/df/GumRotateWhip.html