How to add my stream processing logic in mcu mode.

[zhcf]

trafficstars

I want to process stream with my algorithm before sending stream to clients

One choice is using owt-server analytics mechanaism（https://github.com/open-webrtc-toolkit/owt-server/blob/master/doc/servermd/AnalyticsGuide.md#develop）: my algorithm is opencv algorithm not AI model, so I can not convert it into an openvino model to use the mechanism directly. I need to write a Gstreamer plugin to support my algorithm？

Second choice: whether there are APIs so that I can get stream frame, and use my algorithm processing stream, then send the stream to client?

Thanks

[starwarfan]

1. You can write a Gstreamer plugin and refer to samples to add it in analytics (https://github.com/open-webrtc-toolkit/owt-server/tree/master/source/agent/plugins). To send stream back to clients, you could refer to the sample CPU pipeline and replace those plugins between source and encoder with your own.
2. For experiment, you could get input frame data for analytics agent at onFrame function of GstInternalIn (https://github.com/open-webrtc-toolkit/owt-server/blob/master/source/agent/analytics/videoGstPipeline/GstInternalIn.cpp#L60). It's the entry of analytics pipeline, if you made changes in this function, make sure it won't take a long time and block the pipeline.

[zhcf]

MCU mode， analytics pipeline subscribe one stream in the room, process the stream and retrun result to owt-server. The returned stream will be a new stream to be mixed in the same room?

Thanks

[starwarfan]

If you link analytics pipeline with an output, there will be a new stream in the conference.