track-processors-js
track-processors-js copied to clipboard
Option to preserve aspect ratio and flip on VirtualBackground?
Hello!
Could this support a preserveAspectRatio
and flipImageHorizontal
option to the VideoBackground
transformer?
Getting squashed and flipped images when I try to simply pass through a remote URL to the constructor
VirtualBackground('https://i0.wp.com/apartmentapothecary.com/wp-content/uploads/2021/10/shoe-storage-for-very-narrow-hallway.webp?ssl=1')
^ that's using this image fwiw, notice how it's taller and oriented the other way horizontally
Was messing with the sample, and think this can be achieved pretty simply in BackgroundTransformer
something like this preserves the aspect ratio and flips the image (which is getting flipped by default in the current implementation, for some reason)
async drawVirtualBackground(frame: VideoFrame) {
if (!this.canvas || !this.ctx || !this.segmentationResults || !this.inputVideo) return;
if (this.segmentationResults?.categoryMask) {
this.ctx.filter = 'blur(10px)';
this.ctx.globalCompositeOperation = 'copy';
const bitmap = await maskToBitmap(
this.segmentationResults.categoryMask,
this.segmentationResults.categoryMask.width,
this.segmentationResults.categoryMask.height,
);
this.ctx.drawImage(bitmap, 0, 0, this.canvas.width, this.canvas.height);
this.ctx.filter = 'none';
this.ctx.globalCompositeOperation = 'source-in';
if (this.backgroundImage) {
this.drawImagePreserveAspectRatio(this.backgroundImage, this.canvas.width, this.canvas.height);
} else {
this.ctx.fillStyle = '#00FF00';
this.ctx.fillRect(0, 0, this.canvas.width, this.canvas.height);
}
this.ctx.globalCompositeOperation = 'destination-over';
}
// Draw the frame without flipping
this.ctx.drawImage(frame, 0, 0, this.canvas.width, this.canvas.height);
}
drawImagePreserveAspectRatio(image: ImageBitmap, canvasWidth: number, canvasHeight: number) {
const imgRatio = image.width / image.height;
const canvasRatio = canvasWidth / canvasHeight;
let drawWidth, drawHeight, x, y;
if (imgRatio > canvasRatio) {
drawHeight = canvasHeight;
drawWidth = image.width * (drawHeight / image.height);
x = (canvasWidth - drawWidth) / 2;
y = 0;
} else {
drawWidth = canvasWidth;
drawHeight = image.height * (drawWidth / image.width);
x = 0;
y = (canvasHeight - drawHeight) / 2;
}
// Original flipped version (commented out)
// Apply horizontal flip transformation for background image
this.ctx!.save();
this.ctx!.scale(-1, 1);
this.ctx!.drawImage(image, -x - drawWidth, y, drawWidth, drawHeight);
this.ctx!.restore();
}
Might be nice to do something like
const virtualBackground = VirtualBackground('/background.png', { preserveAspectRatio: true, flipHorizontal: true })