Relationship to WebGPU

[aboba]

Mediacapture-Transform's MSTProcessor interface uses VideoFrame . My understanding is that this represents a GPU buffer, but the relationship to WebGPU's GPUBuffer is unclear to me.

WebGPU, is going into Origin Trial, so it would be useful to be able to try out using Mediacapture-Transform and WebGPU together (e.g. for "Funny Hats" scenarios).

[dogben]

There is quite a bit of discussion in https://github.com/gpuweb/gpuweb/issues/1380 about import.

@Kangz can confirm, but since VideoFrame can now be created from a CanvasImageSource, that covers WebGPU also.

[Kangz]

Yes that's correct. WebGPU can render to canvases so the CanvasImageSource path should work. Maybe additional paths are added in the future but that one should work (at least in terms of spec) today.

Note that GPUBuffer represents unstructured GPU memory (think gpu_malloc) while GPUTexture represents 2D (or 1D / 3D) memory with a specific format in an opaque layout. What we call "buffer" in the media world is closer in concept to a GPUTexture.

[alvestrand]

There's a demo of breakout box / WebGPU interaction at https://webrtc.github.io/samples/src/content/insertable-streams/webgpu/

[dontcallmedom]

related WebGPU integration discussions in https://github.com/webmachinelearning/webnn/issues/226#issuecomment-1010752362

The sample above notes in particular:

  // Using GPUExternalTexture(when it's implemented for Breakout Box frames) will
  // avoid making extra copies through ImageBitmap.

[aboba]

WebCodecs PR 412 proposes conversion of VideoFrame to GPUExternalTexture.