badaix
Zero copying in snapserver on TCP connections for audio data
Is your feature request related to a problem? Please describe.
I played around with zero copying in snapserver on TCP connections for audio data. I've got a working solution and did some measurements.
Describe the solution you'd like
I tackled the problem directly with sendmsg with MSG_ZEROCOPY and a dedicated thread for completion messages (MSG_ERRQUEUE). It's on an exclusive alternative path to the boost:asio stuff (as asio didn't support that) and it's linux only. But fallback is to the (alternative) asio implementation.
My measurements (on my PC with Fedora 42, Intel(R) Core(TM) i7-8700) imply:
- ~15% less CPU load (23% less user CPU load, but system CPU load is increased by 8%)
- ~20% less context switches
- ~90% of the transmitted bytes now use the ZC path (small data and non audio are excluded)
- memory usage and usage pattern seems to be the same than before
Hence the question at hand is: Is that worth the additional ~700 lines of code and the added complexity?
Describe alternatives you've considered
This was just an experiment. Feedback is welcome. I could make a PR of this.
How do you measure CPU load improvements? My snapserver is on GL-MT6000 and with 140 days of uptime it only used 14h of CPU.
Those improvements are always welcome especially in performance constrained environment like routers.
Yes, for newer CPUs the CPU load of snapserver is negligible. Compared to HD video streaming, audio streaming is like a cable modem. Saving 15% on 0.6 may not even be measurable.
I completely agree that this is more interesting for SBCs like raspberry...
And I only implemented ZC in snapserver. It would be possible in snapclient as well. But I haven't tried and hence no numbers...
However I guess that is less interesting on the client. In principle, snapserver could serve multiple streams. But the snapclient has one and only.
However I guess that is less interesting on the client.
It would be absolute interesting! 😀
One of my snapclient is a Ralink RT5350F based HT-MT02!
BusyBox v1.33.2 (2023-03-30 12:18:04 UTC) built-in shell (ash)
_______ ________ __
| |.-----.-----.-----.| | | |.----.| |_
| - || _ | -__| || | | || _|| _|
|_______|| __|_____|__|__||________||__| |____|
|__| W I R E L E S S F R E E D O M
-----------------------------------------------------
OpenWrt 21.02.7, r16847-f8282da11e
-----------------------------------------------------
| Machine: HooToo HT-TM02 |
| Uptime: 98d, 22:27:28 |
| Load: 0.57 0.36 0.27 |
| Flash: total: 2.3MB, free: 1.7MB, used: 25% |
| Memory: total: 27.2MB, free: 10.4MB, used: 61% |
| WAN: |
| LAN: 192.168.1.1 |
| radio0: mode: sta, ssid: XXXX, channel: , conn: 1 |
-----------------------------------------------------
root@HT-TM02-43C ~ # top -b -n 1
Mem: 21900K used, 5924K free, 52K shrd, 1964K buff, 2884K cached
CPU: 0% usr 9% sys 9% nic 81% idle 0% io 0% irq 0% sirq
Load average: 0.98 0.49 0.31 2/37 3447
PID PPID USER STAT VSZ %VSZ %CPU COMMAND
30550 1 root SN 5596 20% 11% /usr/bin/snapclient --logsink system --host snapcast --soundcard iec958:CARD=CODEC,DEV=0
3447 3356 root R 1248 4% 7% top -b -n 1
938 1 root S 4428 16% 0% /usr/sbin/wpa_supplicant -n -s -g /var/run/wpa_supplicant/global
937 1 root S 4332 16% 0% /usr/sbin/hostapd -s -g /var/run/hostapd/global
825 1 root S 1896 7% 0% /sbin/logd -S 256
1000 1 root S 1792 6% 0% /sbin/netifd
1 0 root S 1616 6% 0% /sbin/procd
1056 1 root S 1476 5% 0% /usr/sbin/odhcpd
1381 1 root S 1372 5% 0% /usr/sbin/pingcheck
3356 3355 root S 1256 5% 0% -ash
1326 1 root S< 1252 4% 0% /usr/sbin/ntpd -n -N -S /usr/sbin/ntpd-hotplug -p 0.openwrt.pool.ntp.org -p 1.openwrt.pool.ntp.org -p 2.openwrt.pool.ntp.org -p 3.openwrt.pool.ntp.org
1516 1000 root S 1248 4% 0% udhcpc -p /var/run/udhcpc-wlan0.pid -s /lib/netifd/dhcp.script -f -t 0 -i wlan0 -x hostname:HT-TM02-43C -C -R -O 121
511 1 root S 1244 4% 0% /sbin/ubusd
3355 888 root S 1196 4% 0% /usr/sbin/dropbear -F -P /var/run/dropbear.1.pid -p 22 -K 300 -T 3
888 1 root S 1136 4% 0% /usr/sbin/dropbear -F -P /var/run/dropbear.1.pid -p 22 -K 300 -T 3
546 1 root S 1040 4% 0% /sbin/urngd
512 1 root S 936 3% 0% /sbin/askfirst /usr/libexec/login.sh
339 2 root IW 0 0% 0% [kworker/0:2-eve]
7 2 root SW 0 0% 0% [ksoftirqd/0]
31157 2 root IW 0 0% 0% [kworker/u2:1-ph]
31625 2 root IW 0 0% 0% [kworker/u2:0-ph]
4 2 root IW< 0 0% 0% [kworker/0:0H-kb]
436 2 root SWN 0 0% 0% [jffs2_gcd_mtd8]
135 2 root SW 0 0% 0% [kswapd0]
211 2 root SW 0 0% 0% [spi0]
3 2 root IW 0 0% 0% [kworker/0:0-eve]
634 2 root IW< 0 0% 0% [cfg80211]
116 2 root SW 0 0% 0% [watchdogd]
99 2 root IW< 0 0% 0% [kblockd]
310 2 root IW< 0 0% 0% [ipv6_addrconf]
27 2 root SW 0 0% 0% [oom_reaper]
95 2 root IW< 0 0% 0% [writeback]
2 0 root SW 0 0% 0% [kthreadd]
6 2 root IW< 0 0% 0% [mm_percpu_wq]
97 2 root SW 0 0% 0% [kcompactd0]
313 2 root IW< 0 0% 0% [kworker/0:1H]
root@HT-TM02-43C ~ #
Interesting use case, to what does your openwrt output the music?
I see, snapclient ZC would have one user (at least). However, let me start with the PR for snapserver...
Interesting use case, to what does your openwrt output the music?
Yes it runs snapclient and outputs music to UCA202 USB soundcard! 😎
root@HT-TM02-43C ~ # lsusb
Bus 001 Device 001: ID 1d6b:0002 Linux 5.4.238 ehci_hcd EHCI Host Controller
Bus 002 Device 002: ID 08bb:2902 Burr-Brown from TI USB Audio CODEC
Bus 002 Device 001: ID 1d6b:0001 Linux 5.4.238 ohci_hcd Generic Platform OHCI controller
@xabolcs : I'm just curious, but do you use compression (flac?, other?) or plain PCM?
@aanno
# Default transport codec
# (flac|ogg|opus|pcm)[:options]
# Start Snapserver with "--stream:codec=<codec>:?" to get codec specific options
codec = flac:0
... due to client side resource constraints. 🙃
Well, more than I expected.
FLAC does compress at compression level 0, but it is the lowest level of compression that the format offers. Level 0 still reduces file size considerably compared to the original uncompressed audio
This is a bit OOT, but have you seen https://github.com/badaix/snapcast/discussions/1417 ? It is a question on how to build snapcast for openWRT...
I set up the poll #1442 to get more information about the usage pattern of snapcast.
Dear @xabolcs ,
I've made some experimental (and fundamental!) client changes at my branch controlled-async-loop. It implements excessive ZC. But it also uses a buffer pool and might need more RAM because of that. I've no idea how to compile that for openWRT. But perhaps you want to try...
(I tested it for working but currently haven't done any measurements.)
I've no idea how to compile that for openWRT. But perhaps you want to try...
Sure I'll try, but before that I have to do my package maintainer homework: OpenWrt has v0.28.0 of Snapcast, because of the unreleased "optional SSL" thing.
So thank you for the news, but don't wait for my feedback! 😅