elodin icon indicating copy to clipboard operation
elodin copied to clipboard

Add video streaming and playback to db and editor

Open sphw opened this issue 7 months ago • 0 comments

This PR ended up being pretty big, but all the changes are in support of adding video streaming to the editor. Broadly this PR is made up of the following changes:

Add optional timestamps to msgs When you send the editor a message you can now use the new PacketType::MsgWithTimestamp that allows you to pass a timestamp in

Add fixed rate playback for msgs In order to effectively stream video, we need to ensure that msgs can be replayed. This PR adds a new stream for messages that provides fixed rate playback. In addition I simplified some of the stream logic by removing filtering from real-time and fixed-rate streams. VTableStreams are a strictly better solution when the user needs filtering

Add video stream to editor This PR adds a new tile type that streams video from the db. It only works with H264 right now, because platform support is mixed for AV1 and HVEC. On macOS it decodes video using video-toolbox, the HW accelerated macOS native video api. On Windows and Linux it uses the permissible licensed OpenH264 decoder. In the future we might want to add HW accelerated decoding for Linux and Windows, but that adds a ton of complexity

Add FFMPEG based video streamer

This was my first attempt at setting up streaming video to the db. I used ffmpeg, which turned out to be a real PITA. It works, and is moderately fast, but I reckon we might want to delete this and switch to gstreamer

Add elodinsink gstreamer plugin This is a simple gstreamer plugin that takes a stream of H264 NAL packets, and sends them to the DB

sphw avatar May 13 '25 19:05 sphw