Backing Track Support

[jcard0na]

Just dumping my thoughts on a feature I've had on the back of my head for sometime...

Use Case

Title: Backing Track Files Goal: User copies can play along files stored on the Haxophone Story:

1. User pulls SD card out of Haxophone and inserts on a laptop (Linux, Mac, Windows)
2. User copies .wav or .mp3 files to a dedicated partition on the SD Card
3. User inserts SD card back on Haxophone and boots
4. User enters Accompaniment mode (maybe 1-s draw + A, mnemonic "Accompaniment")
5. User cycles through the different files using the same R1/R2 keys used for changing instruments. Each song begins playing as soon as it is selected.
6. User exits Accompaniment mode mode and the last selected song restarts playing from the beginning.
7. User plays the Haxophone normally while the song is playing. The audio is mixed in real time and heard on the audio out.
8. When the song ends, the user can enter and exit Accompaniment mode (without selecting a different song) to restart the song. Or they can pick a different song. Or they can do nothing and continue playing normally without backing track.

Implementation Notes:

1. Quick experiment to confirm that we can mix in real-time [DONE, 👍]

   1. copy wav file to /boot partition
   2. aplay -D plug:dmix /boot/wikiloops_jam_281585_mix.wav &
   3. Issue the same command again, and hear the second instance mixed with the first

2. Right now the haxophone uses the alsa's hw device. We should use plughw or dmix instead. It is important to confirm that this does not affect latency.

[jcard0na]

Documenting the results of a quick experiment:

Plugging the haxo output directly to dmix fails to initialize fluidsynth with error:

fluidsynth: error: Failed to find an audio format supported by alsa

However, we can connect to dmix via the plug interface as shown below:

--- a/src/alsa.rs
+++ b/src/alsa.rs
@@ -10,7 +10,7 @@ pub fn get_device() -> Result<String, Box<dyn Error>> {
     let cards = Iter::new();
     for c in cards {
         let card = c?;
-        let id_string = format!("hw:{}", card.get_index());
+        let id_string = format!("plug:dmix:{}", card.get_index());
         let card_name = card.get_name()?;
         info!("Found alsa card {}", card_name);
         if card_name == HAXOPHONE_AUDIO_CARD_ID {

The following warnings are raised on init:

fluidsynth: warning: Requested a period size of 64, got 940 instead
fluidsynth: warning: Requested 3 periods, got 4 instead

haxo starts with good audio and horrible latency (feels like 1s), but now we can play at the same time a background track is played with

aplay -D plug:dmix test.wav 
Playing WAVE 'test.wav' : Signed 16 bit Little Endian, Rate 44100 Hz, Stereo

So progress...

[jcard0na]

Tried to bypass the plug device by creating a dmix device with fluidsynth-compatible paramters. Works, but did not improve the latency.

cat ~/.asoundrc
pcm.dmix0 {
	type dmix
	ipc_key 34521
	slave {
    		pcm "hw:0,0"
    		format S16_LE
		channels 2
    		rate 44100
	}
    	hint {
            description "DMix of Card0"
    	}
}

+        let id_string = format!("dmix0");

Init warnings:

fluidsynth: warning: Requested a period size of 64, got 5513 instead

9x the period size translates to 9x latency :cry:

[langeroo]

I was playing around with an idea for a different approach to achieve the same goal. It seemed like it could be more user-friendly/extensible, but so far it does not seem to work.

The thought was:

• set up the RPi as a Bluetooth audio sink
• stream backing track or whatever audio you want from a phone (phone-to-RPi latency is irrelevant since you're synced up on the RPi side of things)
• mix the BT audio with the haxo audio and output to the audio driver

I don't have much experience with audio handling but did get further than I expected; I was able to get bluetooth streaming audio while maintaining the ability to run haxo on its own, but I was unable to mix the two sources. To achieve this, I'm pretty sure haxo needs to send audio to pulseaudio rather than alsa (or at least it seems to be much easier to do it this way, supposedly). When I quickly tested outputting to pulse, the synth was noticeably worse (crackly w/ some latency). I might keep trying to see what I can come up with, but curious @jcard0na if you have tried using pulseaudio instead of alsa before?

[jcard0na]

Hi @langeroo

Very interesting! I have tried pulseaudio in different projects. In my relentless pursuit for low lantency, I found that alsa was unbeatable. That kind of makes sense, as PA sits on top of Alsa (https://askubuntu.com/questions/581128/what-is-the-relation-between-alsa-and-pulseaudio-sound-architecture). That said, I'm am convinced that it is possible to get that mixing to work with low-latency. I just could not figure it out... But someone will!