Add support for one-way audio

[shnhrrsn]

Adds support for turning on audio from the camera to HomeKit.

Off by default, but turned on via an audio: true config param. Requires ffmpeg to be compiled with fdk-aac.

Fixes #33

[adrum]

I tested this out and it works for me, although my audio was a bit choppy. It could just be my hardware.

[jnanes]

Will this be pushed to Npm?

[justme-1968]

+1

[mylesagray]

I spent the last two days or so trying to sort out the choppiness in the audio and I think I've cracked it. The net result is the audio from my fork is now as consistent as the audio I get from my official HomeKit compatible Netatmo Outdoor Camera - just some minor blips, but very usable.

Video demos:

Pre-optimisations: https://www.dropbox.com/s/h7zr2r4qbkqymsd/pre-optimisations.mp4?dl=0

Post-optimisations: https://www.dropbox.com/s/b03c5iwt57vf20d/post-optimisations.mp4?dl=0

I am running a customised version of the fork from here: https://github.com/shnhrrsn/homebridge-camera-ffmpeg-ufv/tree/transmuxing which is part of this PR: https://github.com/gozoinks/homebridge-camera-ffmpeg-ufv/pull/37

Steps to replicate my config, on your homebridge instance:

Uninstall existing homebridge-camera-ffmpeg-ufv plugin, then run:

npm install git://github.com/mylesagray/homebridge-camera-ffmpeg-ufv#transmuxing --save

I opened a PR against your fork @shnhrrsn with my changes included: https://github.com/shnhrrsn/homebridge-camera-ffmpeg-ufv/pull/2

A high level summary of what was changed:

• The -rtsp_transport for ffmpeg was changed to tcp from http as I had seen this made audio smoother with the homebridge-camera-ffmpeg plugin using UniFi cameras.
• Added a dedicated buffer size for the audio, discrete from the bitrate. Through testing abuffsize = 48 seems to give a good smooth stream, it seems to need to be a multiple of the sample rate.
• Changed the audio bitrate from 32K to 24K
• Changed the packet sizes of the video and audio streams to their own individual values - this seemed to make by far the most difference. The packet_size should always be a multiple of 188.
  • Video packet size is set to 1316
  • Audio packet size is set to 188 - this seemed to have to be as low as possible to get a good stream.

Things I couldn't change:

• The HAP specification says that the packet_time (a:ptime) param needs to be set per codec/samplerate - this cannot be done with ffmpeg - as a result we get logs of fragmented or out of time audio packets from the Home.app.

What I've noticed in Console with a filter of any:avconferenced applied, when using the Home.app on macOS to view the streams (equivalent to the iOS streams). Without my changes every time the audio "chops" you see this in the Console logs:

error	16:41:47.275029+0000	avconferenced	 [ERROR] _VCAudioReceiver_ParsePacket:832 FATAL - Too many frames in audio packet, possible corruption

Adding in my changes, you can see in the logs that the SIP audio stream is being interpreted as 16KHz aac and there are no occurrences of the above message.

default	15:16:30.947597+0000	avconferenced	ACMP4AACBaseEncoder.cpp:293:Initialize: (0x7fa4f65b9240) Input format:  1 ch,  16000 Hz, Float32
default	15:16:30.947669+0000	avconferenced	ACMP4AACBaseEncoder.cpp:296:Initialize: (0x7fa4f65b9240) Output format:  1 ch,  16000 Hz, 'aacf' (0x00000000) 0 bits/channel, 0 bytes/packet, 480 frames/packet, 0 bytes/frame
default	15:16:30.948149+0000	avconferenced	ACMP4AACBaseEncoder.cpp:644:Initialize: (0x7fa4f65b9240) @@@@ 'aacf' encoder configuration: srIn = 16000, srOut = 16000, chans = 1, bitRateFormat = 1, bitrate = 24000, quality (complexity) = 64, VBRQ = -1, speechOptimization = 0, packetSizeLimit = 0 (bits), packetBitSizeMin = 0 (bits), mMaxPacketSize = 768 (bytes), userBandwidth = 0, delayMode = 1, mCodecDelay = 240, drcConfiguration = 0, mPrePostFillMask = 0x0
default	15:16:30.948997+0000	avconferenced	ACMP4AACBaseEncoder.cpp:644:Initialize: (0x7fa4f65b9240) @@@@ 'aacf' encoder configuration: srIn = 16000, srOut = 16000, chans = 1, bitRateFormat = 3, bitrate = 20000, quality (complexity) = 64, VBRQ = -1, speechOptimization = 0, packetSizeLimit = 0 (bits), packetBitSizeMin = 0 (bits), mMaxPacketSize = 768 (bytes), userBandwidth = 0, delayMode = 1, mCodecDelay = 240, drcConfiguration = 0, mPrePostFillMask = 0x0
default	15:16:30.949750+0000	avconferenced	ACMP4AACBaseEncoder.cpp:644:Initialize: (0x7fa4f65b9240) @@@@ 'aacf' encoder configuration: srIn = 16000, srOut = 16000, chans = 1, bitRateFormat = 3, bitrate = 20000, quality (complexity) = 64, VBRQ = -1, speechOptimization = 0, packetSizeLimit = 0 (bits), packetBitSizeMin = 0 (bits), mMaxPacketSize = 768 (bytes), userBandwidth = 0, delayMode = 1, mCodecDelay = 240, drcConfiguration = 0, mPrePostFillMask = 0x0

In contrast to my Netatmo Outdoor camera which is natively HomeKit compatible - which uses a 24KHz Opus codec:

default	15:13:43.807176+0000	avconferenced	ACOpusDecoder.cpp:178:Initialize: (0x7fa4f643c640) Input format:  1 ch,  24000 Hz, 'opus' (0x00000000) 0 bits/channel, 0 bytes/packet, 480 frames/packet, 0 bytes/frame
default	15:13:43.807213+0000	avconferenced	ACOpusDecoder.cpp:181:Initialize: (0x7fa4f643c640) Output format:  1 ch,  24000 Hz, Float32

I have had no luck getting Home.app to play audio when changing the codec to Opus, but I haven't tried in earnest.

[jnanes]

would this be merged to the base branch at some point?

[jmbwell]

Is ffmpeg usually compiled with fdk-aac, or will we be asking users to compile their own ffmpeg?

Does this change the base systems (versions of Ubuntu etc) supported?

Is it straightforward to get working on raspberry pi?

[mylesagray]

If we can make it work with Opus 24k rather than AAC-ELD then we will have no need to recompile ffmpeg, as libopus is usually part of ffmpeg bins in most distributions.

As it stands, I ran this on a synology, in a docker container that I built with ffmpeg compiled with the necessary lib for AAC-ELD.

Given the crux of the changes were around packet size for the audio and video rtp streams, that should transfer nicely to Opus, but I haven’t had time to give it a go yet.