pipecat-ai
Image/Video with WebSocket Transports
pipecat version
No response
Python version
No response
Operating System
No response
Question
I've been building an agent using the FastAPIWebsocketTransport. As far as I understand, this only supports audio transmission. Is there any way to integrate image/video functionality (something as simple as the "describe what you see" examples) using WebSocket transports? The way I understand it now, I would either have to pay for a transport service like Daily or Livekit or put in a lot of work to get the WebRTC transport to work. Is it possible to extend the WebSocket transports to support video? Any help or hint would be much appreciated! Thanks.
What I've tried
No response
Context
No response
We have discussed adding a WebsocketTransport to the client SDKs. This is something we plan to do in the near future.
Just an FYI: sending audio and video to a client via websocket is likely to encounter performance issues on real-world networks. We're planning to add it to aid testing for folks building with FastAPIWebsocketTransport so that they can have a web client to do testing with without requiring a different transport.
