viseron icon indicating copy to clipboard operation
viseron copied to clipboard

Published 20 hours ago • verified publisher icon roflcoopter

[Frature Request] Gstreamer audio support on Jetson Nano

Open ksiwy87 opened this issue 2 years ago • 12 comments

Hello,

I would be great if viseron v2 will support sound in streams from cameras by Gstreamer. I have one camera which is adding audio track into the stream despite the fact there is no sound there. Sound track in stream broke Gstreamer right now.

Right now for this camera I'm using ffmpeg which consumes CPU a lot.

ksiwy87 avatar Jun 19 '22 14:06 ksiwy87

@roflcoopter - are you able to add this feature? One of my cameras is not working with gstreamer because of that limitation. The audio is disabled in camera settings, but It still sends an audio track in the video.

ksiwy87 avatar Aug 05 '22 03:08 ksiwy87

Yes soon enough! I spent a lot of time on the last commit that lays the foundation for the component docs.

My plan was to get v2 released before jumping on this but since it's a showstopper I'll try to squeeze it in before release

roflcoopter avatar Aug 05 '22 08:08 roflcoopter

Yes soon enough! I spent a lot of time on the last commit that lays the foundation for the component docs.

My plan was to get v2 released before jumping on this but since it's a showstopper I'll try to squeeze it in before release

Thank you so much!

ksiwy87 avatar Aug 05 '22 15:08 ksiwy87

Do you have any logs on this issue?

roflcoopter avatar Aug 19 '22 15:08 roflcoopter

Hello,

Sorry for delay. Here are the logs: "brama" is this problematic camera: [2022-08-25 23:28:08] [ERROR ] [viseron.components.gstreamer.stream.brama] - NvMMLiteOpen : Block : BlockType = 261 [2022-08-25 23:28:08] [ERROR ] [viseron.components.gstreamer.stream.brama] - NVMEDIA: Reading vendor.tegra.display-size : status: 6 [2022-08-25 23:28:08] [ERROR ] [viseron.components.gstreamer.stream.brama] - NvMMLiteBlockCreate : Block : BlockType = 261 [2022-08-25 23:30:16] [ERROR ] [viseron.components.gstreamer.camera.brama] - Timeout waiting for frame [2022-08-25 23:30:21] [ERROR ] [viseron.watchdog.thread_watchdog] - Thread viseron.camera.brama is dead, restarting [2022-08-25 23:30:21] [ERROR ] [viseron.components.gstreamer.stream.brama] - NvMMLiteOpen : Block : BlockType = 261 [2022-08-25 23:30:21] [ERROR ] [viseron.components.gstreamer.stream.brama] - NVMEDIA: Reading vendor.tegra.display-size : status: 6 [2022-08-25 23:30:21] [ERROR ] [viseron.components.gstreamer.stream.brama] - NvMMLiteBlockCreate : Block : BlockType = 261 [2022-08-25 23:32:31] [ERROR ] [viseron.components.gstreamer.camera.brama] - Timeout waiting for frame [2022-08-25 23:32:36] [ERROR ] [viseron.watchdog.thread_watchdog] - Thread viseron.camera.brama is dead, restarting [2022-08-25 23:32:36] [ERROR ] [viseron.components.gstreamer.stream.brama] - NvMMLiteOpen : Block : BlockType = 261 [2022-08-25 23:32:36] [ERROR ] [viseron.components.gstreamer.stream.brama] - NVMEDIA: Reading vendor.tegra.display-size : status: 6 [2022-08-25 23:32:36] [ERROR ] [viseron.components.gstreamer.stream.brama] - NvMMLiteBlockCreate : Block : BlockType = 261 [2022-08-25 23:34:46] [ERROR ] [viseron.components.gstreamer.camera.brama] - Timeout waiting for frame [2022-08-25 23:34:51] [ERROR ] [viseron.watchdog.thread_watchdog] - Thread viseron.camera.brama is dead, restarting [2022-08-25 23:34:51] [ERROR ] [viseron.components.gstreamer.stream.brama] - NvMMLiteOpen : Block : BlockType = 261 [2022-08-25 23:34:51] [ERROR ] [viseron.components.gstreamer.stream.brama] - NVMEDIA: Reading vendor.tegra.display-size : status: 6 [2022-08-25 23:34:51] [ERROR ] [viseron.components.gstreamer.stream.brama] - NvMMLiteBlockCreate : Block : BlockType = 261 [2022-08-25 23:34:53] [ERROR ] [viseron.components.gstreamer.stream.brama] - reference in DPB was never decoded [2022-08-25 23:34:55] [ERROR ] [viseron.components.gstreamer.stream.brama] - ERROR: from element /GstPipeline:pipeline0/GstSplitMuxSink:mux/GstMP4Mux:mp4mux0: Could not multiplex stream. [2022-08-25 23:34:55] [ERROR ] [viseron.components.gstreamer.stream.brama] - Additional debug info: [2022-08-25 23:34:55] [ERROR ] [viseron.components.gstreamer.stream.brama] - gstqtmux.c(4561): gst_qt_mux_add_buffer (): /GstPipeline:pipeline0/GstSplitMuxSink:mux/GstMP4Mux:mp4mux0: [2022-08-25 23:34:55] [ERROR ] [viseron.components.gstreamer.stream.brama] - Buffer has no PTS. [2022-08-25 23:34:55] [ERROR ] [viseron.components.gstreamer.camera.brama] - GStreamer process has exited [2022-08-25 23:35:00] [ERROR ] [viseron.components.gstreamer.camera.brama] - Restarting frame pipe [2022-08-25 23:35:01] [ERROR ] [viseron.components.gstreamer.stream.brama] - NvMMLiteOpen : Block : BlockType = 261 [2022-08-25 23:35:01] [ERROR ] [viseron.components.gstreamer.stream.brama] - NVMEDIA: Reading vendor.tegra.display-size : status: 6 [2022-08-25 23:35:01] [ERROR ] [viseron.components.gstreamer.stream.brama] - NvMMLiteBlockCreate : Block : BlockType = 261

ksiwy87 avatar Aug 25 '22 21:08 ksiwy87

I added some basic audio support now in the modularize branch, could you test and see if it works for you? Ii should be detected automatically

roflcoopter avatar Sep 10 '22 09:09 roflcoopter

I've updated to the newest version, but the problem is still there.

ksiwy87 avatar Sep 26 '22 16:09 ksiwy87

@roflcoopter - any idea what could be a cause of this problem?

ksiwy87 avatar Nov 06 '22 11:11 ksiwy87

I dont sadly :( Its working for me with my audio camera, however i dont have a Jetson Nano and i cant justify buying one right now.

What does the generated pipeline look for you? Should be printed in the debug logs

roflcoopter avatar Nov 07 '22 08:11 roflcoopter

[2022-12-23 11:05:50] [DEBUG ] [viseron.components.gstreamer.recorder.brama] - Running cleanup [2022-12-23 11:05:50] [DEBUG ] [viseron.components.gstreamer.recorder.brama] - Initializing gstreamer recorder [2022-12-23 11:05:50] [DEBUG ] [viseron.components.gstreamer.camera.brama] - Initializing camera Brama [2022-12-23 11:05:50] [DEBUG ] [viseron.components.gstreamer.stream.brama] - Getting stream information for rtsp://*****:*****@172.22.0.13:554/ [2022-12-23 11:05:50] [DEBUG ] [viseron.components.gstreamer.stream.brama] - FFprobe command: ffprobe -hide_banner -loglevel error -print_format json -show_error -show_streams rtsp://*****:*****@172.22.0.13:554/ [2022-12-23 11:05:52] [INFO ] [viseron.components] - Setup of domain camera for component gstreamer with identifier taras took 2.0 seconds [2022-12-23 11:05:52] [WARNING ] [viseron.components.edgetpu.object_detector.taras] - scan_on_motion_only is enabled but no motion detector is configured. Disabling scan_on_motion_only [2022-12-23 11:05:52] [INFO ] [viseron.components] - Setup of domain object_detector for component edgetpu with identifier taras took 0.1 seconds [2022-12-23 11:05:52] [INFO ] [viseron.components.nvr.nvr.taras] - Motion detector is disabled [2022-12-23 11:05:52] [INFO ] [viseron.components.nvr.nvr.taras] - NVR for camera Taras initialized [2022-12-23 11:05:52] [INFO ] [viseron.components] - Setup of domain nvr for component nvr with identifier taras took 0.0 seconds [2022-12-23 11:05:52] [INFO ] [viseron.components] - Setup of domain camera for component gstreamer with identifier plac took 2.3 seconds [2022-12-23 11:05:52] [WARNING ] [viseron.components.edgetpu.object_detector.plac] - scan_on_motion_only is enabled but no motion detector is configured. Disabling scan_on_motion_only [2022-12-23 11:05:52] [INFO ] [viseron.components] - Setup of domain object_detector for component edgetpu with identifier plac took 0.0 seconds [2022-12-23 11:05:52] [INFO ] [viseron.components.nvr.nvr.plac] - Motion detector is disabled [2022-12-23 11:05:52] [INFO ] [viseron.components.nvr.nvr.plac] - NVR for camera Plac initialized [2022-12-23 11:05:52] [INFO ] [viseron.components] - Setup of domain nvr for component nvr with identifier plac took 0.0 seconds [2022-12-23 11:05:52] [INFO ] [viseron.components] - Setup of domain camera for component gstreamer with identifier schody took 2.5 seconds [2022-12-23 11:05:52] [WARNING ] [viseron.components.edgetpu.object_detector.schody] - scan_on_motion_only is enabled but no motion detector is configured. Disabling scan_on_motion_only [2022-12-23 11:05:52] [INFO ] [viseron.components] - Setup of domain object_detector for component edgetpu with identifier schody took 0.0 seconds [2022-12-23 11:05:52] [INFO ] [viseron.components.nvr.nvr.schody] - Motion detector is disabled [2022-12-23 11:05:52] [INFO ] [viseron.components.nvr.nvr.schody] - NVR for camera Schody initialized [2022-12-23 11:05:52] [INFO ] [viseron.components] - Setup of domain nvr for component nvr with identifier schody took 0.0 seconds [2022-12-23 11:05:52] [ERROR ] [viseron.components.gstreamer.stream.taras] - NvMMLiteOpen : Block : BlockType = 261 [2022-12-23 11:05:52] [ERROR ] [viseron.components.gstreamer.stream.taras] - NVMEDIA: Reading vendor.tegra.display-size : status: 6 [2022-12-23 11:05:52] [ERROR ] [viseron.components.gstreamer.stream.taras] - NvMMLiteBlockCreate : Block : BlockType = 261 [2022-12-23 11:05:53] [ERROR ] [viseron.components.gstreamer.stream.plac] - NvMMLiteOpen : Block : BlockType = 261 [2022-12-23 11:05:53] [ERROR ] [viseron.components.gstreamer.stream.plac] - NVMEDIA: Reading vendor.tegra.display-size : status: 6 [2022-12-23 11:05:53] [ERROR ] [viseron.components.gstreamer.stream.plac] - NvMMLiteBlockCreate : Block : BlockType = 261 [2022-12-23 11:05:53] [ERROR ] [viseron.components.gstreamer.stream.schody] - NvMMLiteOpen : Block : BlockType = 261 [2022-12-23 11:05:53] [ERROR ] [viseron.components.gstreamer.stream.schody] - NVMEDIA: Reading vendor.tegra.display-size : status: 6 [2022-12-23 11:05:53] [ERROR ] [viseron.components.gstreamer.stream.schody] - NvMMLiteBlockCreate : Block : BlockType = 261 [2022-12-23 11:05:55] [DEBUG ] [viseron.components.gstreamer.stream.brama] - Stream information from FFprobe: Width: 2560 Height: 1440 FPS: 25.0 Video Codec: h264 Audio Codec: pcm_mulaw [2022-12-23 11:05:55] [DEBUG ] [viseron.components.gstreamer.camera.brama] - Resolution: 2560x1440 @ 20 FPS [2022-12-23 11:05:55] [DEBUG ] [viseron.components.gstreamer.camera.brama] - Camera Brama initialized [2022-12-23 11:05:55] [INFO ] [viseron.components] - Setup of domain camera for component gstreamer with identifier brama took 5.6 seconds [2022-12-23 11:05:55] [WARNING ] [viseron.components.edgetpu.object_detector.brama] - scan_on_motion_only is enabled but no motion detector is configured. Disabling scan_on_motion_only [2022-12-23 11:05:55] [INFO ] [viseron.components] - Setup of domain object_detector for component edgetpu with identifier brama took 0.0 seconds [2022-12-23 11:05:55] [DEBUG ] [viseron.components.nvr.nvr.brama] - Initializing NVR for camera Brama [2022-12-23 11:05:55] [INFO ] [viseron.components.nvr.nvr.brama] - Motion detector is disabled [2022-12-23 11:05:55] [DEBUG ] [viseron.components.nvr.nvr.brama] - Waiting for first frame [2022-12-23 11:05:55] [DEBUG ] [viseron.components.gstreamer.camera.brama] - Starting capture thread [2022-12-23 11:05:55] [DEBUG ] [viseron.components.gstreamer.stream.brama] - GStreamer decoder command: gstreamer_brama --gst-debug-level=1 -q rtspsrc location=rtsp://*****:*****@172.22.0.13:554/ name=input_stream do-timestamp=true latency=0 protocols=tcp ! rtpjitterbuffer do-lost=true drop-on-latency=true ! rtph264depay ! tee name=depayed_stream ! queue ! nvv4l2decoder enable-max-performance=true drop-frame-interval=10 ! nvvidconv ! video/x-raw,format=(string)NV12 ! fdsink depayed_stream. ! queue ! h264parse ! splitmuxsink async-finalize=true send-keyframe-requests=true max-size-bytes=0 name=mux muxer=mp4mux max-size-time=5000000000 location=/segments/brama/%01d.mkv input_stream. ! queue ! decodebin ! audioconvert ! queue ! voaacenc ! mux.audio_0 [2022-12-23 11:05:55] [INFO ] [viseron.components.nvr.nvr.brama] - NVR for camera Brama initialized [2022-12-23 11:05:55] [INFO ] [viseron.components] - Setup of domain nvr for component nvr with identifier brama took 0.0 seconds [2022-12-23 11:05:55] [INFO ] [viseron.core] - Viseron initialized in 6.8 seconds [2022-12-23 11:05:56] [ERROR ] [viseron.components.gstreamer.stream.brama] - NvMMLiteOpen : Block : BlockType = 261 [2022-12-23 11:05:56] [ERROR ] [viseron.components.gstreamer.stream.brama] - NVMEDIA: Reading vendor.tegra.display-size : status: 6 [2022-12-23 11:05:56] [ERROR ] [viseron.components.gstreamer.stream.brama] - NvMMLiteBlockCreate : Block : BlockType = 261 [2022-12-23 11:08:04] [ERROR ] [viseron.components.gstreamer.camera.brama] - Timeout waiting for frame [2022-12-23 11:08:04] [DEBUG ] [viseron.components.gstreamer.camera.brama] - Stopping capture thread [2022-12-23 11:08:09] [DEBUG ] [viseron.components.gstreamer.camera.brama] - Timed out trying to stop camera. Killing pipe [2022-12-23 11:08:09] [ERROR ] [viseron.watchdog.thread_watchdog] - Thread viseron.camera.brama is dead, restarting [2022-12-23 11:08:09] [DEBUG ] [viseron.components.gstreamer.camera.brama] - Starting capture thread [2022-12-23 11:08:09] [DEBUG ] [viseron.components.gstreamer.stream.brama] - GStreamer decoder command: gstreamer_brama --gst-debug-level=1 -q rtspsrc location=rtsp://*****:*****@172.22.0.13:554/ name=input_stream do-timestamp=true latency=0 protocols=tcp ! rtpjitterbuffer do-lost=true drop-on-latency=true ! rtph264depay ! tee name=depayed_stream ! queue ! nvv4l2decoder enable-max-performance=true drop-frame-interval=10 ! nvvidconv ! video/x-raw,format=(string)NV12 ! fdsink depayed_stream. ! queue ! h264parse ! splitmuxsink async-finalize=true send-keyframe-requests=true max-size-bytes=0 name=mux muxer=mp4mux max-size-time=5000000000 location=/segments/brama/%01d.mkv input_stream. ! queue ! decodebin ! audioconvert ! queue ! voaacenc ! mux.audio_0 [2022-12-23 11:08:10] [ERROR ] [viseron.components.gstreamer.stream.brama] - NvMMLiteOpen : Block : BlockType = 261 [2022-12-23 11:08:10] [ERROR ] [viseron.components.gstreamer.stream.brama] - NVMEDIA: Reading vendor.tegra.display-size : status: 6 [2022-12-23 11:08:10] [ERROR ] [viseron.components.gstreamer.stream.brama] - NvMMLiteBlockCreate : Block : BlockType = 261 [2022-12-23 11:10:19] [ERROR ] [viseron.components.gstreamer.camera.brama] - Timeout waiting for frame [2022-12-23 11:10:19] [DEBUG ] [viseron.components.gstreamer.camera.brama] - Stopping capture thread [2022-12-23 11:10:24] [DEBUG ] [viseron.components.gstreamer.camera.brama] - Timed out trying to stop camera. Killing pipe [2022-12-23 11:10:24] [ERROR ] [viseron.watchdog.thread_watchdog] - Thread viseron.camera.brama is dead, restarting [2022-12-23 11:10:24] [DEBUG ] [viseron.components.gstreamer.camera.brama] - Starting capture thread [2022-12-23 11:10:24] [DEBUG ] [viseron.components.gstreamer.stream.brama] - GStreamer decoder command: gstreamer_brama --gst-debug-level=1 -q rtspsrc location=rtsp://*****:*****@172.22.0.13:554/ name=input_stream do-timestamp=true latency=0 protocols=tcp ! rtpjitterbuffer do-lost=true drop-on-latency=true ! rtph264depay ! tee name=depayed_stream ! queue ! nvv4l2decoder enable-max-performance=true drop-frame-interval=10 ! nvvidconv ! video/x-raw,format=(string)NV12 ! fdsink depayed_stream. ! queue ! h264parse ! splitmuxsink async-finalize=true send-keyframe-requests=true max-size-bytes=0 name=mux muxer=mp4mux max-size-time=5000000000 location=/segments/brama/%01d.mkv input_stream. ! queue ! decodebin ! audioconvert ! queue ! voaacenc ! mux.audio_0 [2022-12-23 11:10:25] [ERROR ] [viseron.components.gstreamer.stream.brama] - NvMMLiteOpen : Block : BlockType = 261 [2022-12-23 11:10:25] [ERROR ] [viseron.components.gstreamer.stream.brama] - NVMEDIA: Reading vendor.tegra.display-size : status: 6 [2022-12-23 11:10:25] [ERROR ] [viseron.components.gstreamer.stream.brama] - NvMMLiteBlockCreate : Block : BlockType = 261

ksiwy87 avatar Dec 23 '22 10:12 ksiwy87

@roflcoopter - Any idea why it's not working?

ksiwy87 avatar Feb 06 '23 12:02 ksiwy87

No i have no idea :( I do not have a Nano to run any tests so i would need someone to figure out how the pipeline should look in order to implement it.

This is the command that we have now which doesnt seem to be working on the Nano:

gstreamer_brama --gst-debug-level=1 -q rtspsrc location=rtsp://*****:*****@172.22.0.13:554/ name=input_stream do-timestamp=true latency=0 protocols=tcp ! rtpjitterbuffer do-lost=true drop-on-latency=true ! rtph264depay ! tee name=depayed_stream ! queue ! nvv4l2decoder enable-max-performance=true drop-frame-interval=10 !  nvvidconv ! video/x-raw,format=(string)NV12 ! fdsink depayed_stream. ! queue ! h264parse ! splitmuxsink async-finalize=true send-keyframe-requests=true max-size-bytes=0 name=mux muxer=mp4mux max-size-time=5000000000 location=/segments/brama/%01d.mkv input_stream. ! queue ! decodebin ! audioconvert ! queue ! voaacenc ! mux.audio_0

roflcoopter avatar Feb 08 '23 13:02 roflcoopter