VideoFrameProcessor example doesn't seem to work

[RaymondFJ]

What are you trying to do?

I'm trying to implement a video frame processor on API: amazon-chime-sdk-v24.min.js in order to crop/resize a screen share

How can the documentation be improved to help your use case?

The documentation provided at: https://aws.github.io/amazon-chime-sdk-js/modules/videoprocessor.html

does not seem complete in that it specifies to use a deviceID string (it seems that passing the object instead is required) and that the example itself does not work; when replacing buffers in the process function; it does not seem to modify the local screen share output (and I would assume the remote share as well)

the passed in canvas does indeed contain the video frame; but it seems nothing can be done to modify it from there.

What documentation have you looked at so far?

https://aws.github.io/amazon-chime-sdk-js/modules/videoprocessor.html

[XHatan]

hey @RaymondFJ , are you trying to send screen capture stream through chooseVideoInputDevice or startContentShareFromScreenCapture ?

when replacing buffers in the process function; it does not seem to modify the local screen share output (and I would assume the remote share as well)

Are you using VideoFrameProcssorPipeline and set the screen captured stream to it ?

[RaymondFJ]

Hello XHatan thank you for responding! I will start by saying I am not very experienced with the chime API so it is quite likely I am doing something wrong.

when I start my screen share I run:

const contentShareStream = await session.audioVideo.startContentShareFromScreenCapture(); const stages = [new VideoResizeProcessor(4/3)]; // constructs processor const transformDevice = new ChimeSDK.DefaultVideoTransformDevice( session.logger, contentShareStream, // docs say it should be device id string, but I find that throws an error and the object gets me further stages);

await session.audioVideo.chooseVideoInputDevice(transformDevice);

this prompts me to share my screen; after doing so my observer code gets called: which I use to start getting local output into a video tag.

observer = { videoTileDidUpdate: async tileState => { // Ignore a tile without attendee ID and videos. if (!tileState.boundAttendeeId || !tileState.isContent) { return; }

				  const yourAttendeeId = session.configuration.credentials.attendeeId;

				  const boundAttendeeId = tileState.boundAttendeeId;

				  const baseAttendeeId = new ChimeSDK.DefaultModality(boundAttendeeId).base();
				  if (baseAttendeeId === yourAttendeeId) {
					ChimeSDK.DefaultVideoTile.connectVideoStreamToVideoElement(tileState.boundVideoStream, $('#video-0')[0], false);
				  }
				},		
    };

My processor, process function is called regularly and passed the proper canvas with the screen share; but replacing/returning the buffer with my own as per the example seems to have no effect on the local video share.

function VideoResizeProcessor(){ this.targetCanvas = document.createElement('canvas');

$('body').append(this.targetCanvas);
this.targetCanvasCtx = this.targetCanvas.getContext('2d');
this.canvasVideoFrameBuffer = new ChimeSDK.CanvasVideoFrameBuffer(this.targetCanvas);

this.renderWidth = 0;
this.renderHeight = 0;
this.sourceWidth = 0;
this.sourceHeight = 0;

}

VideoResizeProcessor.prototype.process = async function(buffers){ const canvas = buffers[0].asCanvasElement(); const frameWidth = canvas.width; const frameHeight = canvas.height;

  // error handling to skip resizing
  if (frameWidth === 0 || frameHeight === 0) {
	return buffers;
  }

  // re-calculate the cropped width and height

  // copy the frame to the intermediate canvas
  this.targetCanvasCtx.drawImage(canvas, 0, 0, 250, 250, 0, 0, 250, 250);

  // replace the video frame with the resized one for subsequent processor
  buffers[0] = this.canvasVideoFrameBuffer;
  return buffers;

}

Thanks in advance for any info you can provide!

[kjslough]

I am also trying to get this to work for frame by frame processing of a screen share content stream. The sample code seems geared towards processing a webcam stream.

For the "typical workflow" described in the docs as:

1. Create an array of custom VideoFrameProcessors.
2. Create a VideoTransformDevice from a Device and the array of VideoFrameProcessors.
3. Call meetingSession.audioVideo.chooseVideoInputDevice with the VideoTransformDevice.

it seems like step 3 is what you would do for a webcam stream.

For a screen share stream, it is not clear what the typical workflow would be. Steps 1 and 2 are the same. For step 3, one would start by calling startContentShareFromScreenCapture and getting a MediaStream object in return. But what is the next step? Am I supposed to create a DefaultVideoFrameProcessorPipeline and call setInputMediaStream, passing in my screen share media stream? And then what? Or something else entirely?

It would be great if the docs could cover both the scenario of modifying the webcam stream AND a screen share stream.

[XHatan]

Hey, @RaymondFJ @kjslough

Thanks for providing the information. Currently the APIs only support injection of the video processor for video calling VideoInputDevice chooseVideoInputDevice is for video calling and not for content sharing.

For content sharing, the APIs do not support injection of video processor.

For experiment purposes, you can try workaround using DefaultVideoFrameProcessorPipeline for content sharing

• get hold of a MediaStream in your application without Chime APIs (e.g through https://developer.mozilla.org/en-US/docs/Web/API/MediaDevices/getDisplayMedia)
• Construct DefaultVideoFrameProcessorPipeline with VideoProcessors
• Call DefaultVideoFrameProcessorPipeline.setInputMediaStream with the originalMediaStream and then call getActiveOutputMediaStream to get transformed MediaStream
• at the end of content sharing, call destroy on DefaultVideoFrameProcessorPipeline after you call "stopContentShare"
• clean up the original MediaStream in order to stop capture.

 let pipe = new DefaultVideoFrameProcessorPipeline(logger, [new EmojifyVideoFrameProcessor("Test")]);

await pipe.setInputMediaStream(mediaStream);
        
const newStream = pipe.getActiveOutputMediaStream();

this.audioVideo.startContentShare(mediaStream);

release

this.audioVideo.stopContentShare();
pipe.destroy();

for (const track of mediaStream.getTracks()) {
    track.stop();
}

The callbacks should be the same as you would use content sharing.

Hope this would help.

[XHatan]

Hey @RaymondFJ @kjslough any update ?

[RaymondFJ]

Thanks for getting back to me; haven't been able to get back to this task yet. Hopefully I will have some time this week and will report back on my findings; thanks again!

On Tue, Feb 16, 2021 at 3:20 PM Xuan He [email protected] wrote:

Hey @RaymondFJ https://github.com/RaymondFJ @kjslough https://github.com/kjslough any update ?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/aws/amazon-chime-sdk-js/issues/1047#issuecomment-780093719, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADAH6GILLW3ZAZL74AMC4JTS7LHQBANCNFSM4XDFHKCA .

[kjslough]

Yes, thanks for the info. If this feature is always going to be "experimental" then there is not much point investigating it further for use in a real world product. Is it likely that the SDK will support this "officially" at some point?

[RaymondFJ]

@XHatan Thank you again for the example code!

I was able to get the example working with one exception that in your example 'this.audioVideo.startContentShare(mediaStream);' would pass newStream instead of mediaStream.

I was able to crop the video stream, though I did find some strange issues with the height of the reported video frame canvas size; it seemed as though the bottom 60 pixels or so was filled with random/stale/offscreen video data; my guess is the frame isn't cropping correctly/accounting for the 'you are sharing this tab' bar that chrome drops down; but it is entirely possible that there is a bug in my code.

Hopefully this feature can be added to the SDK as I'd be worried that it would cease to work since you mentioned it was experimental.

[vidya-mohan]

@RaymondFJ - While we don't have a first class API to transform- here is a code snippet to unblock you. See https://github.com/aws/amazon-chime-sdk-js/issues/1731#issuecomment-954302701

[hensmi-amazon]

Resolving as we are not likely going to add a specific API for this anytime soon. I wouldn't call the workaround for content share experimental ;P, I just released changes to our demo and guides that do basically what was described above: https://github.com/aws/amazon-chime-sdk-js/pull/2467 .