track-processors-js icon indicating copy to clipboard operation
track-processors-js copied to clipboard

Option to preserve aspect ratio and flip on VirtualBackground?

Open sgrund14 opened this issue 5 months ago • 0 comments

Hello!

Could this support a preserveAspectRatio and flipImageHorizontal option to the VideoBackground transformer?

Getting squashed and flipped images when I try to simply pass through a remote URL to the constructor VirtualBackground('https://i0.wp.com/apartmentapothecary.com/wp-content/uploads/2021/10/shoe-storage-for-very-narrow-hallway.webp?ssl=1')

Screenshot 2024-09-30 at 7 32 36 PM

^ that's using this image fwiw, notice how it's taller and oriented the other way horizontally

Was messing with the sample, and think this can be achieved pretty simply in BackgroundTransformer

something like this preserves the aspect ratio and flips the image (which is getting flipped by default in the current implementation, for some reason)

async drawVirtualBackground(frame: VideoFrame) {
    if (!this.canvas || !this.ctx || !this.segmentationResults || !this.inputVideo) return;
    if (this.segmentationResults?.categoryMask) {
      this.ctx.filter = 'blur(10px)';
      this.ctx.globalCompositeOperation = 'copy';
      const bitmap = await maskToBitmap(
        this.segmentationResults.categoryMask,
        this.segmentationResults.categoryMask.width,
        this.segmentationResults.categoryMask.height,
      );
      this.ctx.drawImage(bitmap, 0, 0, this.canvas.width, this.canvas.height);
      this.ctx.filter = 'none';
      this.ctx.globalCompositeOperation = 'source-in';
      if (this.backgroundImage) {
        this.drawImagePreserveAspectRatio(this.backgroundImage, this.canvas.width, this.canvas.height);
      } else {
        this.ctx.fillStyle = '#00FF00';
        this.ctx.fillRect(0, 0, this.canvas.width, this.canvas.height);
      }

      this.ctx.globalCompositeOperation = 'destination-over';
    }
    // Draw the frame without flipping
    this.ctx.drawImage(frame, 0, 0, this.canvas.width, this.canvas.height);
  }

drawImagePreserveAspectRatio(image: ImageBitmap, canvasWidth: number, canvasHeight: number) {
    const imgRatio = image.width / image.height;
    const canvasRatio = canvasWidth / canvasHeight;
    let drawWidth, drawHeight, x, y;

    if (imgRatio > canvasRatio) {
      drawHeight = canvasHeight;
      drawWidth = image.width * (drawHeight / image.height);
      x = (canvasWidth - drawWidth) / 2;
      y = 0;
    } else {
      drawWidth = canvasWidth;
      drawHeight = image.height * (drawWidth / image.width);
      x = 0;
      y = (canvasHeight - drawHeight) / 2;
    }
    
    // Original flipped version (commented out)
    // Apply horizontal flip transformation for background image
    this.ctx!.save();
    this.ctx!.scale(-1, 1);
    this.ctx!.drawImage(image, -x - drawWidth, y, drawWidth, drawHeight);
    this.ctx!.restore();
  }

Might be nice to do something like

const virtualBackground = VirtualBackground('/background.png', { preserveAspectRatio: true, flipHorizontal: true })

sgrund14 avatar Sep 30 '24 23:09 sgrund14