EyeGazeCorrection.

[riju]

Why ?

We have all heard “eyes are the window to the soul” and their importance in effective communication. The disparity of locations of the subject and the camera make it hard to have eye contact during the video call. Recent consumer-level platforms have been able to solve the eye gaze correction problem, more often employing custom AI accelerators on the client platforms. The ability to render the gaze corrected face would help in a realistic imitation of real-world communication in an increasingly virtual world and undoubtedly be a welcome feature for the WebRTC developer community, something native platforms have been offering for some time.

Microsoft eloquently blogged about EyeGazeCorrection for their Surface lineup. MediaFoundation already has a KSCAMERA_EXTENDEDPROP_EYEGAZECORRECTION_ON starting from Windows 11, provided there is driver support.

Apple’s FaceTime already has something very similar in the form of Attention Correction on iOS 14.0 or later devices.

How ?

Strawman Proposal

<script>
const videoStream = await navigator.mediaDevices.getUserMedia({
   video: true,
});

// Show camera video stream to the user.
const video = document.querySelector("video");
video.srcObject = videoStream;

// Get video track capabilities.
const videoTrack = videoStream.getVideoTracks()[0];
const capabilities = videoTrack.getCapabilities();

// Check whether eyegazeCorrection is supported.
if (!capabilities.eyegazeCorrection) return;

async function applyEyegazeCorrection() {
  try {
    await track.applyConstraints({
      eyegazeCorrection: true;
    });
  } catch (err) {
    console.error(err);
  }
}
</script>