Most efficient way of broadcasting a still image?

[leroypijlman]

Describe the issue We want our bot to display a different logo based on the context of the meeting. So we will not be able to use the pre-configured 'Application image' that you can set in Azure. That's why we want to broadcast the logo (a still image) as video output. Up until this point, we have only used the audio socket. We're slightly worried about the performance implications the addition of a video channel may bring.

Questions

• What are the performance implications of broadcasting a still image?
• If just an image is continuously streamed, does the data have to be retransmitted?
• Is there a way for us to configure the SDK to transmit the data at a lower frequency? (e.g. configuring it to be 1 fps.)
• Do you have any other guidelines or suggestions to do this as efficiently as possible?

[InDieTasten]

What are the performance implications of broadcasting a still image?

Should be pretty low impact, because the encoder will have an easy time to encode a non-changing image, which also results in low increase of bitrate needed by the bot.

If just an image is continuously streamed, does the data have to be retransmitted?

Yes, it needs to be a continuous stream of the still image. Newly connecting clients will request key frames from your video feed in order to start displaying your stream.

Is there a way for us to configure the SDK to transmit the data at a lower frequency? (e.g. configuring it to be 1 fps.)

Not sure about that one. I know receiving streams are VFR (variable frame rate), but I'm not sure if sending arbitrary framerates works. I wouldn't rule it out though.

Do you have any other guidelines or suggestions to do this as efficiently as possible?

You might be able to pre-encode a loop of your different logos to h264 and stream those into the meeting, skipping the continuous encoding during the meetings.

Another idea might be to use the actual bot registration logo and have multiple of those. If you have a fixed number it might be easier to just have multiple registrations. This obviously comes with overhead for running multiple bots (for different identities) and managing consent for all of them.

---

Further notes: I don't believe there is an API to broadcast video streams as a service-hosted bot. So running as an application hosted media bot is a requirement.

[yk-recall]

Hi @leroypijlman, it sounds like you have a situation where you want to show different logos per meeting and I think your concern for performance implications are generally correct. I'd say that if you're looking to just stream an image though, it probably won't be as expensive as you might expect and wanna double down on @InDieTasten answer there

If you're still concerned about how expensive this would be, you could try streaming raw video buffers directly into the meeting. To transmit video at 1 FPS efficiently using the AudioVideoPlaybackBot in a Microsoft Teams meeting, you could preprocess your video to 1 FPS with H.264 encoding at ~100-200 kbps (using ffmpeg) and sending raw video buffers, or modify the bot’s video pipeline to sample one frame per second. Then you could configure the bot’s VideoSocket to send these frames via WebSockets

That said, there is a lot of maintenance and issues that can go wrong with this method. Instead, you could try using a third party solution like Recall.ai which is a universal API for deploying meeting bots. Sending a bot to a meeting and outputting a customer logo is as simple as a post request.

Here is the API from Recall to display an image on the bot: https://docs.recall.ai/docs/output-video-in-meetings Here is the API from Recall to display video on a bot if you wanted to show something more dynamic: https://docs.recall.ai/docs/stream-media

Let me know if you have any questions about this!

[leroypijlman]

Thanks for both your answers, I really appreciate it. It helped me make some quick strides. It is an Application Hosted Media bot.

You can apparently configure the frame rate when initializing your media session.

new VideoSocketSettings()
                    {
                      StreamDirections  = StreamDirection.Sendonly,
                      MediaType = MediaType.Video,
                      SupportedSendVideoFormats = new List<VideoFormat> {
                        VideoFormat.H264_320x180_1_875Fps
                      },
                      // CroppingMode = CroppingMode.FitToFrame,
                    },

This is as small as I could make it, but should serve it's purpose. I managed to make a start and get a quick version up and running.

It works, however (aside from not being entirely sure if I took the correct approach) I ran into an issue:

The streamed video is ONLY visible for participants that are in the call when _videoSocket.Send() is called. All participants that join subsequently do not see the image, but just a blank screen.

Here's a very minimal example of what I'm doing to broadcast the video:

public class VideoService
{
    private readonly IVideoSocket _videoSocket;
    private readonly byte[] currentFrameData;
    private MediaSendStatus videoMediaSendStatus = MediaSendStatus.Inactive;

    public VideoService(IVideoSocket videoSocket)
    {
        currentFrameData = System.IO.File.ReadAllBytes("C:/test/test.h264");
        _videoSocket = videoSocket;
        videoSocket.VideoSendStatusChanged += (sender, args) => { videoMediaSendStatus = args.MediaSendStatus; };
        videoSocket.VideoKeyFrameNeeded += (sender, args) =>
        {
            if (videoMediaSendStatus == MediaSendStatus.Active)
            {
                SendVideoBuffer();
            }
        };
    }

    private void SendVideoBuffer()
    {
        var buffer = new VideoSendBuffer(currentFrameData, (uint)currentFrameData.Length,
            VideoFormat.H264_320x180_1_875Fps);
        _videoSocket.Send(buffer);
    }
}

VideoService() is then just instantiated in my CallHandler after the call has been initialized.

The source video is a single frame h264 encoded video created with ffmpeg as @yk-recall somewhat suggested:

ffmpeg -loop 1 -framerate 1.875 -i test.png -vf "scale=320:180" -c:v libx264 -profile:v baseline -tune stillimage -x264-params keyint=1 -pix_fmt yuv420p -t 1.6 -f h264 test.h264

My question now is: Why is the visual not shown for any subsequent joining participants? The logs show a continuous stream of video frames being emitted, so the frames are being rebroadcasted.

[InDieTasten]

Great that it works as far as it does. Regarding the remaining issue, I would try the "naive" approach first, which would be to let the SDK itself encode the video on the fly and see how that does.

The logs that suggest continous emission of frames are from the SDK, not your own logging correct? You should try to diagnose how often your SendVideoBuffer method is being invoked. I suspect the SDK might repeat p frames from your h264 source indefinitely.