Detect Orientation from Peer Recording/Video Stream

[daryl3]

RecordRTC is successfully saving the video streams as webm/video files. However, portrait videos from WebRTC Peers are rotated sideways. Some are rotated 90 degrees clockwise and others are rotated 90 degrees counter clockwise, based on the device used. My iPhone 6 Plus is always rotated 90 degrees counter clockwise, other devices are rotated 90 degrees clockwise, etc.

So a portrait video stream that is 640x480 when displayed ends up being recorded sideways at a 480x640 video.

Its a simple enough procedure to rotate the completed video the correct direction using FFMpeg, but the recorded file does not have the "rotate" tag in the metadata. So I am unable to determine which direction I should rotate my video.

I have been unable to detect the stream orientation from the stream, the video element or the completed recording.

Thank you!

[yogaboy]

I'm having the same issue. Is there any reliable way to detect the orientation from the peer? Perhaps the peer can broadcast over the data channel it's current orientation when recording starts? Using the Beta from ondeviceorientation events, if the Beta is high, the device is portrait, otherwise it is landscape.

I'm not sure if there is any other way in the tagging. Obviously the browser knows the rotation needed to show the remote stream correctly and is handling it internally...

[farizyoga]

i am having same trouble, any idea for this ?

[yogaboy]

We're not alone in this: https://stackoverflow.com/questions/49565739/webrtc-detect-orientation-from-peer-video-stream

My only thought, since this is from a remote peer is to use the data track to broadcast changes in the device orientation. We would just need to be able to specify an orientation change to match the difference between the interpreted frame capture and the real orientation.

[daryl3]

I posted that particular stack overflow question. I never got an answer from the community. What I ended up doing was asking the peer (using a data channel) what its camera orientation is and saving it along with the stream chunk. I was never pleased with this answer, but it was all I was ever able to come up with.

[yogaboy]

@daryl3 can I ask what you did to configure the size and orientation for the RecordRTC config? I can certainly use the RTC data channel to pass this along before starting to record, but don't know where to configure this for the recording.

[daryl3]

@yogaboy I used the data channel to ask the remote connection what its device dimensions and orientation were. Then I created a javascript array of the information about each remote connection. Every 5 seconds, I dump a blob of the information into a database and store the information I have about the connection. Once the recording is done, I take all the blobs and re-assemble them into a single .webm file. Which I then convert to the format I want using FFMPeg. I use the information I stored about the connection in the database to re-orient video using FFMpeg. The only problem I ended up having is if the video is really short like 1 second or so, and I hadn't yet received a response from the peer about the orientation, etc. Hope this helps you.

[yogaboy]

Thanks @daryl3. Unfortunately, my process is a little different, and the source webm is already messed up if it's portrait mode on the device. RecordRTC takes the captures with portrait sizing, but only uses the landscape height for the frame, tiling the top of the video in the remaining vertical space.

See below:

[daryl3]

@yogaboy We experienced this same issue with the video stream being all bad. It only happened on iOS video feeds. Android video feeds were working just fine. We tracked it down to being a problem with Google's WebRTC builds, that we were using in iOS. We build Google's WebRTC Manually. We found that version 68 was working correctly. But versions 69-71 all had the problem your describing. I don't know about the Coco Pods build versions. We reverted to build 68 and plan to look into more of why it is happening later and submit it to Google. Also, it could just be that Google changed something in their WebRTC SDK and didn't update their examples on how to use it. Something with the rendering settings, etc. However, we just reverted and haven't looked into it beyond that. I plan to go through the commits and and find out exactly what commit broken it at some point.

[yogaboy]

@muaz-khan Any thoughts on how we can configure recording to force the correct orientation and aspect ratio? While we can get information from the RTC peer, I don't know where I would alter the config to ensure this corrected information is applied for the recording capture.

[yutou15]

Thanks @daryl3. Unfortunately, my process is a little different, and the source webm is already messed up if it's portrait mode on the device. RecordRTC takes the captures with portrait sizing, but only uses the landscape height for the frame, tiling the top of the video in the remaining vertical space.

See below:

I got the same issue on Mac. But I found it OK on Windows. It seems to be caused by a bug in the MacBook Chrome.

[cthompso1993]

FYI for anyone that stumbles upon this... it's to do with the SDP offer that gets sent to participants. Found an old fix on another site that seems to fix this (with a slight modification)

Part of the SDP offer is something like: a=extmap:4 urn:3gpp:video-orientation Conveniently, if you remove this line from the SDP offer in transition, it solves the overlapping video above as well as another issue I was having, that the recorded video was rotated despite neither occupant rotating their device.

Note that the extmap number (4 above) can change, so the fix for me was to basically loop through the numbers 1->20 and remove that line if it was there (could probably do this with regex instead).

Hope this helps someone in the future!

[phzio]

This is fantastic news. We are using Twilio's SDK so I will inform them of this and see if they can apply it to their framework.

Thank you so much!

[hktalent]

I posted that particular stack overflow question