maverick
maverick copied to clipboard
goodrobots
Add web based realtime video
https://developer.mozilla.org/en-US/Apps/Fundamentals/Audio_and_video_delivery/Live_streaming_web_audio_and_video
https://github.com/video-dev/hls.js http://videojs.com/ https://www.isrv.pw/html5-live-streaming-with-mpeg-dash https://github.com/arut/nginx-rtmp-module http://nginx-rtmp.blogspot.co.uk/ https://coaxion.net/blog/2013/10/streaming-gstreamer-pipelines-via-http/ https://coaxion.net/blog/2014/05/http-adaptive-streaming-with-gstreamer/ https://github.com/131/h264-live-player https://github.com/mbebenita/Broadway
Hls/Dash features chunked playlist video streaming with rich javascript frontends. Unfortunately it looks like hls/dash are inherently unsuitable for realtime video, and are far more complex to implement. Initially, implement a much simpler http video just to get something working. h264 in javascript looks promising.
RTSP plugin for chrome works well with visiond: https://www.videoexpertsgroup.com/vxg-chrome-plugin/
Looks like https://github.com/131/h264-live-player is likely to be along the right route, but needs websockets and server implementation. Therefore bumping this to Maverick 1.2 target and will look to implement in web ui. For now, add a simple page with the vxg-chrome-plugin support.
@fnoop For what its worth, I have also had a look in this space previously and ended up here: https://github.com/131/h264-live-player. What are your thoughts on running the JS server on the platform to stream video?
@SamuelDudley Yes I looked at that, but from a quick glance it looks like it converts the rtsp/udp stream to websockets/tcp at the server end. The advantage of the vxg method is that it does that conversion at the browser end, so the stream between the uav and the ground browser remains udp (at least that's what it looked like at first glance). This one uses a similar method: https://github.com/Streamedian/html5_rtsp_player/wiki/HTML5-RTSP-Player So none of these are actually using rtsp between the server and the client except for the vxg method, but rather are transmuting or encapsulating in websockets (to my limited understanding). Looked at webrtc as well but that's more complex at the server end and not supported by ios. Suspect that when we look at the webui proper, we'll end up implementing the websockets proxy method, unless we can come up with way of doing it on the browser in javascript or MSE.
I think it's spot on to offer easy to use video preview.
Wish this was open source: https://www.linux-projects.org/uv4l/
Also suggest this come with a caution to the user. It would only take a few client connects to saturate your real world WiFi link if offering a 720 or 1080 stream. 4G connect might be a bit better than wifi but I bet it would not take much to trip QoS upload limits for most service providers. I have no hard data handy but there are calculators like this one: http://www.stardot.com/bandwidth-and-storage-calculator
For GCS env it would might be good to have roadmap for ways to handle sharing?
https://janus.conf.meetecho.com/index.html https://jitsi.org/
Good description of webrtc session: https://levelup.gitconnected.com/build-your-own-video-chat-with-vue-webrtc-socketio-node-redis-eb51b78f9f55
Implement WebRTC. Good support now been merged into gstreamer: https://opensource.com/article/19/1/gstreamer https://github.com/centricular/gstwebrtc-demos Note webrtc on gstreamer has funky features like FEC that made a big difference in wifibroadcast. https://gstreamer.freedesktop.org/documentation/webrtc/index.html?gi-language=c https://gstreamer.freedesktop.org/data/events/gstreamer-conference/2017/Matthew%20Waters%20-%20GStreamer%20WebRTC.pdf http://blog.nirbheek.in/2018/02/gstreamer-webrtc.html
So is this going to be LAN only. No need for STUN or TURN servers whilst handling the signaling locally as described in the vue demo link you provided back in August? I think that works if you only want one way video with option to go to a relay server for those who want to get it out on WAN. But WebRTC makes my head hurt.
WebRTC is peer to peer, very low latency. There are no intermediate/proxy servers. It also allows either uni or bi-directional video along with a host of other features. I think it's routable so can be either 'LAN' or 'WAN' - although not sure those distinctions are relevant for this case. It's a real headache to understand - unnecessarily complicated - but once we get a recipe worked out then I suspect this will be the 'ultimate' solution. Hardware accelerated support is also built-in to pretty much every desktop and mobile browser as well by now.
My understanding is that the data communication is peer to peer but the STUN / TURN servers are needed for discovery and messaging.
https://github.com/centricular/gstwebrtc-demos/blob/master/sendrecv/gst/webrtc-sendrecv.py might be a good place to start
from what i understand if we keep it to LAN it's not nearly as bad. yes, STUN/TURN required to move out to WAN and traverse firewall/NAT.
also read that webRTC is complicated because companies make money providing STUN/TURN services. someday perhaps we can do a dApp but don't think we need to run our own private blockchain just yet.
https://stackoverflow.com/questions/45613981/is-webrtc-without-any-server-not-even-a-signaling-server-possible
Good description of webrtc session: https://levelup.gitconnected.com/build-your-own-video-chat-with-vue-webrtc-socketio-node-redis-eb51b78f9f55
provides signaling example in that demo.
Oh thank god, sample python code. I can't tell you how painful working out gst rtsp python code was in visiond..
and we may want to support full webRTC in future as one of the features of dronesense and other similar platforms is to allow text chat to help teams comm over distance
I'm hoping we can use webrtc to relay video as well - kind of like meshed networking. An obvious application would be where you have a 'mothership' higher-altitude relay UAV like a circling long-endurance plane, that a bunch of other UAVs connect to for control, video and telemetry relay. The mothership would then act as a 'server' that clients on the ground can connect to, to retrieve/display the video and telemetry, and relay control commands to the UAVs.
The hen / chick model you mention is important for SAR and is pretty high up my list of interests.
I guess first step is to get the example python code running...
This will only work maverick to maverick correct? Just based on fact you will need to roll your own signaling so you need same client both sides.
Kind of - as I understand it there is no 'standard' webrtc client. But we should be able to make literally a single file webpage that could be served anywhere - even publicly - that people could use to connect to the backends.
gstreamer webrtcbin is now available:
[dev] [mav@maverick-ubuntuvm ~/code/vid]$ gst-inspect-1.0 webrtcbin
Factory Details:
Rank primary (256)
Long-name WebRTC Bin
Klass Filter/Network/WebRTC
Description A bin for webrtc connections
Author Matthew Waters <[email protected]>
Plugin Details:
Name webrtc
Description WebRTC plugins
Filename /srv/maverick/software/gstreamer/lib/gstreamer-1.0/libgstwebrtc.so
Version 1.16.2
License LGPL
Source module gst-plugins-bad
Source release date 2019-12-03
Binary package GStreamer Bad Plug-ins source release
Origin URL Unknown package origin
However python gobject introspection can't find the typelibs:
[dev] [mav@maverick-ubuntuvm ~/code/vid]$ python3 webrtc-sendrecv.py
Traceback (most recent call last):
File "webrtc-sendrecv.py", line 11, in <module>
gi.require_version('Gst', '1.0')
File "/srv/maverick/software/python/lib/python3.7/site-packages/gi/__init__.py", line 129, in require_version
raise ValueError('Namespace %s not available' % namespace)
ValueError: Namespace Gst not available
On binary installs (eg. raspberry or tegra platform), webrtcbin is available but the typelibs aren't:
[dev] [mav@maverick-nano ~/code/vid]$ python3 webrtc-sendrecv.py
Traceback (most recent call last):
File "webrtc-sendrecv.py", line 13, in <module>
gi.require_version('GstWebRTC', '1.0')
File "/srv/maverick/software/python/lib/python3.7/site-packages/gi/__init__.py", line 129, in require_version
raise ValueError('Namespace %s not available' % namespace)
ValueError: Namespace GstWebRTC not available
Looks like we need to re-compile plugins_bad (which contains webrtc) even on binary platforms as they don't seem to include the typelib. Either that or copy/hack the typelib GstWebRTC-1.0.typelib into place manually on the binary platforms, if that works.
eg. on jetson nano the typelibs from binary packages live in /usr/lib/aarch64-linux-gnu/girepository-1.0. Copy GstWebRTC-1.0.typelib from a source compiled platform (maverick-ubuntuvm) into here and try.
[dev] [mav@maverick-nano ~/code/vid]$ python3 webrtc-sendrecv.py
Missing gstreamer plugins: ['nice']
It does indeed seem to work, so perhaps that's an easier method, even if it's not for the same version (it's from compiled version 1.16.2, nano binary install is 1.14.5).
Binary package gir1.2-gst-plugins-bad-1.0 provides the GSTWebRTC introspection, gstreamer1.0-nice provides the nice plugin.
[dev] [mav@maverick-nano ~/code/vid]$ python3 webrtc-sendrecv.py
usage: webrtc-sendrecv.py [-h] [--server SERVER] peerid
webrtc-sendrecv.py: error: the following arguments are required: peerid
progress!
Using the server https://webrtc.nirbheek.in/, we can connect and negotiate successfully using webrtc-sendrecv.py:
[dev] [mav@maverick-nano ~/code/vid]$ python3 webrtc-sendrecv.py 7463
Sending offer:
v=0
o=- 2335782948630536004 0 IN IP4 0.0.0.0
s=-
t=0 0
a=ice-options:trickle
a=msid-semantic:WMS sendrecv
m=video 9 UDP/TLS/RTP/SAVPF 97
c=IN IP4 0.0.0.0
a=setup:actpass
a=ice-ufrag:g9kMX3yktuHT343et1SaD9OweDklEMa3
a=ice-pwd:0bsj7Jr5oUZh1QRTS5B6er8Ey7o496nV
a=sendrecv
a=rtcp-mux
a=rtcp-rsize
a=rtpmap:97 VP8/90000
a=rtcp-fb:97 nack pli
a=framerate:30
a=ssrc:1109946848 msid:user332872450@host-d718308c webrtctransceiver0
a=ssrc:1109946848 cname:user332872450@host-d718308c
a=mid:video0
a=fingerprint:sha-256 96:6A:99:BA:1C:6B:40:DE:DE:6C:C8:76:3C:0F:E9:4B:C8:B2:C2:57:1C:73:E7:A4:E4:B7:EC:75:17:D3:80:FB
m=audio 9 UDP/TLS/RTP/SAVPF 96
c=IN IP4 0.0.0.0
a=setup:actpass
a=ice-ufrag:s3npkgATtq0mylghjmrQjZHqBNCYimFi
a=ice-pwd:Zj084uEQklS0Om2aDNCjIjqYkW42muDy
a=sendrecv
a=rtcp-mux
a=rtcp-rsize
a=rtpmap:96 OPUS/48000/2
a=rtcp-fb:96 nack pli
a=fmtp:96 sprop-maxcapturerate=48000;sprop-stereo=0
a=ssrc:117495130 msid:user332872450@host-d718308c webrtctransceiver1
a=ssrc:117495130 cname:user332872450@host-d718308c
a=mid:audio1
a=fingerprint:sha-256 96:6A:99:BA:1C:6B:40:DE:DE:6C:C8:76:3C:0F:E9:4B:C8:B2:C2:57:1C:73:E7:A4:E4:B7:EC:75:17:D3:80:FB
Received answer:
v=0
o=- 8882333815724943527 2 IN IP4 127.0.0.1
s=-
t=0 0
a=msid-semantic: WMS
m=video 9 UDP/TLS/RTP/SAVPF 97
c=IN IP4 0.0.0.0
a=rtcp:9 IN IP4 0.0.0.0
a=ice-ufrag:VDxi
a=ice-pwd:H+K/CGZ/tW7I9X9cgJ3AzLTp
a=ice-options:trickle
a=fingerprint:sha-256 3F:AF:AB:3E:52:BC:13:51:29:47:50:69:83:56:2E:01:32:E1:20:4C:C0:17:5B:E2:E9:41:C7:90:73:90:B5:46
a=setup:active
a=mid:video0
a=recvonly
a=rtcp-mux
a=rtcp-rsize
a=rtpmap:97 VP8/90000
a=rtcp-fb:97 nack pli
m=audio 9 UDP/TLS/RTP/SAVPF 96
c=IN IP4 0.0.0.0
a=rtcp:9 IN IP4 0.0.0.0
a=ice-ufrag:4aCx
a=ice-pwd:q4z/ljCBqd07mfLKlv9ukXcw
a=ice-options:trickle
a=fingerprint:sha-256 3F:AF:AB:3E:52:BC:13:51:29:47:50:69:83:56:2E:01:32:E1:20:4C:C0:17:5B:E2:E9:41:C7:90:73:90:B5:46
a=setup:active
a=mid:audio1
a=recvonly
a=rtcp-mux
a=rtpmap:96 OPUS/48000/2
a=fmtp:96 minptime=10;useinbandfec=1
So now need to work out how to serve video locally through a webpage.
Switched to use Janus-gateway as webrtc proxy in the short-medium term. This acts as a media proxy from gstreamer stream and also takes care of all the signalling. Janus provides a bunch of sample javascript for the browser signalling, or we can use a webpack node module like: https://github.com/TechTeamer/janus-api Here is an example of how to consume it through vue.js: https://gist.github.com/Informatic/fc318c7ce58b33975356183800456e7d This is worth trying to start with as it's a cleaner design.