Support server-based rendering via streaming video?

[ianmackenzie]

Rough idea: have Elm app running on the same machine as a Rust executable doing high-performance rendering with Vulkan, real-time ray tracing etc.; stream current scene description from Elm to Rust, and stream back rendering result as video.

[ianmackenzie]

Conversation with @danabrams on possible approaches:

ianmackenzie [10:51 AM] Hey Dan - hope you had fun in Oslo! Looking forward to watching the various talks How much experience have you had with the server side of streaming video?

danabrams [10:52 AM] I have some experience. I’ve never built anything on the scale of Wowza, but I’ve built custom hls segmenters and things like encoders and spent lots of time with ffmpeg. Yes.

ianmackenzie [10:53 AM] I was brainstorming about ways to do really complex) high performance 3D graphics in Elm, and one idea I had was to send a scene description to something like a Rust backend which would render the scene and then stream the result back to the browser as video Was trying to figure out how that could be built most easily

danabrams [10:53 AM] Yeah, well within my wheelhouse. Pretty easy to do ffmpeg/libav (or gstreamer).

ianmackenzie [10:54 AM] Like, would you do the encoding directly in Rust, or use something like https://github.com/arut/nginx-rtmp-module and just send it raw image data to process/compress... Use ffmpeg as a library, you mean?

danabrams [10:55 AM] Yes, that’s what nginx rtmp is doing. How important is compatibility across browsers to you?

ianmackenzie [10:56 AM] The idea is that it would be interactive, so you could orbit a model by mouse/touch in the browser Compatibility is not super important

danabrams [10:56 AM] Oh, wait, no, I have it. Gstreamer just got a webrtc module. You can receive it in the browser via webrtc, the. You don’t have to worry about the latency of other formats.

ianmackenzie [10:56 AM] I'm thinking of this as a replacement for desktop CAD programs

danabrams [10:57 AM] So the issue is going to be latency. There are on demand game streaming programs, but latency is an issue. But for CAD it may be acceptably low.

ianmackenzie [10:57 AM] I'm ok having the server run on the same machine as the client =)

danabrams [10:57 AM] Gstreamer has excellent rust support and also can take OpenGL contexts as a source.

ianmackenzie [10:57 AM] Oohhh that sounds perfect

danabrams [10:59 AM] I don’t know if fmp4 will work, that’d be simpler, but webrtc definitely will. I think hls/dash will have too much latency, even locally.

ianmackenzie [11:00 AM] Basically I'm thinking of distributing a Rust executable which you would start up on your own machine, then it would serve up an Elm-based UI but handle the heavy rendering (with multithreading, Vulkan, other stuff not available in the browser) itself Ok, great insights, thanks!

danabrams [11:00 AM] But I have probably an easier solution, if it’s always local: use ffmpeg or gstreamer to convert the OpenGL context to a v4l2 device. Make it look like a webcam.

ianmackenzie [11:00 AM] Intriguing!

danabrams [11:01 AM] That’s probably the lowest latency, highest quality, lowest cpu usage solution. It can even send raw frames, instead of compressing first.

ianmackenzie [11:01 AM] What is "v4l2" exactly?

danabrams [11:02 AM] Video 4 Linux 2. It’s a standardized video capture format.

ianmackenzie [11:02 AM] ...does it work on Windows? Or is it possible to do something similar on Windows? (edited)

danabrams [11:03 AM] Yes. It’s a standard now. Windows has a separate standard as well that may be superior, if you only need windows support. That’s why we call it v4l2 instead of video 4 Linux.

ianmackenzie [11:03 AM] Yeah, I'm thinking of tools that would be distributed within my company Which is pretty much exclusively Windows But it would be ideal to set it up in a cross platform way so anyone can use it

danabrams [11:04 AM] Well, you could do it with f-sharp using media foundation or directshow instead. But yeah, ffmpeg and chrome/Firefox should be all you need. I can actually test it for you pretty easily in a week or two if you want.

ianmackenzie [11:08 AM] Sounds like a lot of good options! Just wrote a bunch of them down in my notebook to read up on

danabrams [11:08 AM] If v4l2 doesn’t work on windows, ffmpeg also supports dshow.