mediacapture-extensions icon indicating copy to clipboard operation
mediacapture-extensions copied to clipboard

Add background segmentation mask

Open eehakkin opened this issue 1 month ago • 1 comments

Hi!

This adds capabilities, constraints and settings for background segmentation mask. Those are fairly obvious.

For the feature to be useful, the actual background segmentation mask must be provided to web apps. There are various ways to do that:

  1. In my PoC, I changed the stream of video frames to be a stream of interleaved background segmentation mask and real video frames and extended video frame metadata with a background segmentation mask flag so that web apps can tell segmentation mask and real video frames apart. However, that makes it awkward to process such streams and very unclear how to encode them.
  2. In this PR, the real video frame and background segmentation mask frame are bundled together which simplifies processing of the streams and allows encoders to encode real video frames normally. The background segmentation mask frames for their part are mostly for local consumption only.
  3. Another option would be to utilize an alpha channel. However, there are problems with that approach:
    • Some pixel formats (such as NV12 and NV21) do not have corresponding alpha channel formats. So it would not be possible to such add an alpha channel and then later to drop it in order to get the original frame. Instead of that, the whole frame would have to be converted to a difference format.
    • There are no canvas composite operations, for instance, to operate with alpha masks whereas they work great with grayscale masks.
    • Pixels which are certainly background would be completely transparent. For completely transparent pixel the color is practically irrelevant and some compression algorithms could group all completely transparent pixel together and thus lose color information.

/cc @riju


Preview | Diff

eehakkin avatar May 08 '24 22:05 eehakkin