mediacapture-extensions icon indicating copy to clipboard operation
mediacapture-extensions copied to clipboard

Face Detection: How metadata should be tied to MediaStreamTrack video frames

Open youennf opened this issue 2 years ago • 5 comments

Following on https://github.com/w3c/mediacapture-extensions/pull/69 and media capture transform, face detection metadata could be made available to mediastreamtrack transforms. There are a few possibilities we could envision. The following come to mind:

  1. Attach FaceDetection metadata to VideoFrame with dedicated face detection metadata getter/setter (new VideoFrame slot that can be cloned/postMessaged).
  2. Attach FaceDetection metadata to VideoFrame using a generic metadata mechanism (mechanism to be defined, see VideoTrackGenerator).
  3. Make MediaStreamTrack transforms expose objects that have a VideoFrame and a metadata object.
  4. Extend MediaStreamTrackProcessor.readable with a face detection metadata getter (related to the last read video frame). And VideoTrackGenerator.writable with a metadata setter.

youennf avatar Aug 19 '22 12:08 youennf