RustAudio
Android won't adhere to fixed buffer size
Hi all. I've created this pretty simple program that just listens to audio from the microphone and plays it back out to the speakers. All it does is record a sample, pass it to a ring buffer, and the output callback will copy that to the output.
Take a look:
pub fn loopback_audio() {
const BUFFER_SIZE: usize = 480;
const RINGBUFFER_SIZE: usize = BUFFER_SIZE * 2;
let (mut prod, mut cons) = RingBuffer::<f32>::new(RINGBUFFER_SIZE);
let host = cpal::default_host();
let input_device = host.default_input_device().unwrap();
let output_device = host.default_output_device().unwrap();
let input_config = cpal::StreamConfig {
channels: 1,
sample_rate: cpal::SampleRate(48000),
buffer_size: cpal::BufferSize::Default
};
let output_config: cpal::StreamConfig = input_config.clone();
let input_stream = input_device
.build_input_stream(
&input_config,
move |data: &[f32], _: &_| {
if let Ok(chunk) = prod.write_chunk_uninit(BUFFER_SIZE) {
chunk.fill_from_iter(data.to_owned()); // push to ring buffer
}
},
move |err| {
eprintln!("There was an input error: {:?}", err);
},
None,
)
.unwrap();
let output_stream = output_device
.build_output_stream(
&output_config,
move |data: &mut [f32], _: &cpal::OutputCallbackInfo| {
debug!("Entered callback");
let available = cons.slots();
let to_read = data.len().min(available);
match cons.read_chunk(to_read) {
Ok(chunk) => {
debug!("Read a chunk from the ringbuffer");
let (first, second) = chunk.as_slices(); //sort of like first half, second half, because rb is circular
let mid = first.len();
data[..mid].copy_from_slice(first);
if second.len() != 0 {
data[mid..].copy_from_slice(second);
}
debug!("Copied all");
chunk.commit_all();
debug!("Committed all");
}
Err(error) => debug!("{} ", error),
}
},
move |err| {
debug!("There was an output error: {:?}", err);
},
None,
)
.unwrap();
input_stream.play().unwrap();
output_stream.play().unwrap();
std::thread::sleep(Duration::from_secs(150));
}
However, when I run this, I can see that it runs a few times, but eventually crashes with this error message:
Launching lib/main.dart on Pixel 6 in debug mode...
Resolving dependencies...
Downloading packages...
args 2.4.2 (2.5.0 available)
collection 1.18.0 (1.19.0 available)
github 9.17.0 (9.24.0 available)
http 1.1.0 (1.2.2 available)
http_parser 4.0.2 (4.1.0 available)
path 1.8.0 (1.9.0 available)
petitparser 5.4.0 (6.0.2 available)
toml 0.14.0 (0.16.0 available)
version 3.0.0 (3.0.2 available)
Got dependencies!
9 packages have newer versions incompatible with dependency constraints.
Try `dart pub outdated` for more information.
Compiling bin/build_tool_runner.dart to kernel file bin/build_tool_runner.dill.
INFO: Precompiled binaries are disabled
INFO: Building rust_lib_flutter_rust_test for aarch64-linux-android
INFO: Building rust_lib_flutter_rust_test for i686-linux-android
INFO: Building rust_lib_flutter_rust_test for x86_64-linux-android
✓ Built build/app/outputs/flutter-apk/app-debug.apk
Connecting to VM Service at ws://127.0.0.1:57042/zUXjnBVwEb0=/ws
I/OboeAudio(18953): openStreamInternal() INPUT -------- OboeVersion1.8.1 --------
D/OboeAudio(18953): AAudioLoader(): dlopen(libaaudio.so) returned 0x3c43ad50f5ab366f
I/AAudio (18953): AAudioStreamBuilder_openStream() called ----------------------------------------
I/AudioStreamBuilder(18953): rate = 48000, channels = 1, channelMask = 0x80000001, format = 5, sharing = SH, dir = INPUT
I/AudioStreamBuilder(18953): device = 0, sessionId = -1, perfMode = 10, callback: ON with frames = 0
I/AudioStreamBuilder(18953): usage = 1, contentType = 2, inputPreset = 6, allowedCapturePolicy = 0
I/AudioStreamBuilder(18953): privacy sensitive = false, opPackageName = (null), attributionTag = (null)
D/AudioStreamBuilder(18953): build() MMAP not used because AAUDIO_PERFORMANCE_MODE_LOW_LATENCY not requested.
D/utter_rust_test(18953): PlayerBase::PlayerBase()
D/AAudioStream(18953): setState(s#1) from 0 to 2
I/AAudio (18953): AAudioStreamBuilder_openStream() returns 0 = AAUDIO_OK for s#1 ----------------
D/OboeAudio(18953): AudioStreamAAudio.open() format=2, sampleRate=48000, capacity = 2880
D/OboeAudio(18953): calculateDefaultDelayBeforeCloseMillis() default = 21
D/OboeAudio(18953): AudioStreamAAudio.open: AAudioStream_Open() returned AAUDIO_OK = 0
I/OboeAudio(18953): openStreamInternal() OUTPUT -------- OboeVersion1.8.1 --------
I/AAudio (18953): AAudioStreamBuilder_openStream() called ----------------------------------------
I/AudioStreamBuilder(18953): rate = 48000, channels = 1, channelMask = 0x80000001, format = 5, sharing = SH, dir = OUTPUT
I/AudioStreamBuilder(18953): device = 0, sessionId = -1, perfMode = 10, callback: ON with frames = 0
I/AudioStreamBuilder(18953): usage = 1, contentType = 2, inputPreset = 6, allowedCapturePolicy = 0
I/AudioStreamBuilder(18953): privacy sensitive = false, opPackageName = (null), attributionTag = (null)
D/AudioStreamBuilder(18953): build() MMAP not used because AAUDIO_PERFORMANCE_MODE_LOW_LATENCY not requested.
D/utter_rust_test(18953): PlayerBase::PlayerBase()
D/AudioStreamTrack(18953): open(), request notificationFrames = 0, frameCount = 0
D/AAudioStream(18953): setState(s#2) from 0 to 2
I/AAudio (18953): AAudioStreamBuilder_openStream() returns 0 = AAUDIO_OK for s#2 ----------------
D/OboeAudio(18953): AudioStreamAAudio.open() format=2, sampleRate=48000, capacity = 3848
D/OboeAudio(18953): calculateDefaultDelayBeforeCloseMillis() default = 41
D/OboeAudio(18953): AudioStreamAAudio.open: AAudioStream_Open() returned AAUDIO_OK = 0
D/AAudio (18953): AAudioStream_requestStart(s#1) called --------------
D/AAudioStream(18953): setState(s#1) from 2 to 3
D/AAudio (18953): AAudioStream_requestStart(s#1) returned 0 ---------
D/AAudio (18953): AAudioStream_requestStart(s#2) called --------------
D/AAudioStream(18953): setState(s#2) from 2 to 3
D/AudioStreamLegacy(18953): onAudioDeviceUpdate(deviceId = 22)
D/AudioStreamLegacy(18953): onAudioDeviceUpdate(deviceId = 22)
D/AAudio (18953): AAudioStream_requestStart(s#2) returned 0 ---------
D/rust_lib_flutter_rust..(18953): Entered callback
D/rust_lib_flutter_rust..(18953): Read a chunk
D/rust_lib_flutter_rust..(18953): Copied all
D/rust_lib_flutter_rust..(18953): Committed all
D/rust_lib_flutter_rust..(18953): Entered callback
D/rust_lib_flutter_rust..(18953): Read a chunk
D/rust_lib_flutter_rust..(18953): Copied all
D/rust_lib_flutter_rust..(18953): Committed all
D/AudioStreamLegacy(18953): onAudioDeviceUpdate(deviceId = 3)
D/AAudioStream(18953): setState(s#1) from 3 to 4
D/rust_lib_flutter_rust..(18953): Entered callback
D/rust_lib_flutter_rust..(18953): Read a chunk
D/rust_lib_flutter_rust..(18953): Copied all
D/rust_lib_flutter_rust..(18953): Committed all
D/AAudioStream(18953): setState(s#2) from 3 to 4
D/rust_lib_flutter_rust..(18953): Entered callback
D/rust_lib_flutter_rust..(18953): Read a chunk
D/rust_lib_flutter_rust..(18953): Copied all
D/rust_lib_flutter_rust..(18953): Committed all
D/rust_lib_flutter_rust..(18953): Entered callback
D/rust_lib_flutter_rust..(18953): Read a chunk
D/rust_lib_flutter_rust..(18953): Copied all
D/rust_lib_flutter_rust..(18953): Committed all
D/rust_lib_flutter_rust..(18953): Entered callback
D/rust_lib_flutter_rust..(18953): Read a chunk
F/libc (18953): Fatal signal 6 (SIGABRT), code -1 (SI_QUEUE) in tid 19092 (AudioTrack), pid 18953 (utter_rust_test)
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
Build fingerprint: 'google/oriole/oriole:14/AP2A.240605.024/11860263:user/release-keys'
Revision: 'MP1.0'
ABI: 'arm64'
Timestamp: 2024-07-27 11:45:32.841996546-0600
Process uptime: 4s
Cmdline: com.example.flutter_rust_test
pid: 18953, tid: 19092, name: AudioTrack >>> com.example.flutter_rust_test <<<
uid: 10341
tagged_addr_ctrl: 0000000000000001 (PR_TAGGED_ADDR_ENABLE)
signal 6 (SIGABRT), code -1 (SI_QUEUE), fault addr --------
x0 0000000000000000 x1 0000000000004a94 x2 0000000000000006 x3 0000007556b749d0
x4 0000007440dc95a8 x5 0000007440dc95a8 x6 0000007440dc95a8 x7 0000000000000000
x8 00000000000000f0 x9 0000007820e25350 x10 0000000000000001 x11 0000007820e76170
x12 0000000000000003 x13 0000000000000012 x14 0000000000000000 x15 000000000000000c
x16 0000007820edcfd0 x17 0000007820ec8560 x18 000000742d5e4000 x19 0000000000004a09
x20 0000000000004a94 x21 00000000ffffffff x22 0000000000000000 x23 0000007440dfed68
x24 0000007556b74c70 x25 0000007440c626d8 x26 0000007828ceb780 x27 0000000000000000
x28 0000000000000b40 x29 0000007556b74a50
lr 0000007820e5f8b8 sp 0000007556b749b0 pc 0000007820e5f8e4 pst 0000000000001000
28 total frames
backtrace:
#00 pc 000000000005d8e4 /apex/com.android.runtime/lib64/bionic/libc.so (abort+164) (BuildId: 1d36f8ae6e0af6158793abea7d4f4f2b)
#01 pc 00000000001b4e98 /data/app/~~t_t-ZausPkFT7OqxtUcQ4A==/com.example.flutter_rust_test-6UUUaPpOY4r5WiJSCaJqNg==/lib/arm64/librust_lib_flutter_rust_test.so
#02 pc 00000000001b29a0 /data/app/~~t_t-ZausPkFT7OqxtUcQ4A==/com.example.flutter_rust_test-6UUUaPpOY4r5WiJSCaJqNg==/lib/arm64/librust_lib_flutter_rust_test.so
#03 pc 00000000001b2888 /data/app/~~t_t-ZausPkFT7OqxtUcQ4A==/com.example.flutter_rust_test-6UUUaPpOY4r5WiJSCaJqNg==/lib/arm64/librust_lib_flutter_rust_test.so
#04 pc 00000000001b25b4 /data/app/~~t_t-ZausPkFT7OqxtUcQ4A==/com.example.flutter_rust_test-6UUUaPpOY4r5WiJSCaJqNg==/lib/arm64/librust_lib_flutter_rust_test.so
#05 pc 00000000001b105c /data/app/~~t_t-ZausPkFT7OqxtUcQ4A==/com.example.flutter_rust_test-6UUUaPpOY4r5WiJSCaJqNg==/lib/arm64/librust_lib_flutter_rust_test.so
#06 pc 00000000001b2340 /data/app/~~t_t-ZausPkFT7OqxtUcQ4A==/com.example.flutter_rust_test-6UUUaPpOY4r5WiJSCaJqNg==/lib/arm64/librust_lib_flutter_rust_test.so
#07 pc 00000000001cde48 /data/app/~~t_t-ZausPkFT7OqxtUcQ4A==/com.example.flutter_rust_test-6UUUaPpOY4r5WiJSCaJqNg==/lib/arm64/librust_lib_flutter_rust_test.so
#08 pc 00000000001d04c4 /data/app/~~t_t-ZausPkFT7OqxtUcQ4A==/com.example.flutter_rust_test-6UUUaPpOY4r5WiJSCaJqNg==/lib/arm64/librust_lib_flutter_rust_test.so
#09 pc 00000000000c5018 /data/app/~~t_t-ZausPkFT7OqxtUcQ4A==/com.example.flutter_rust_test-6UUUaPpOY4r5WiJSCaJqNg==/lib/arm64/librust_lib_flutter_rust_test.so
#10 pc 00000000000bda7c /data/app/~~t_t-ZausPkFT7OqxtUcQ4A==/com.example.flutter_rust_test-6UUUaPpOY4r5WiJSCaJqNg==/lib/arm64/librust_lib_flutter_rust_test.so
#11 pc 00000000000b1ff0 /data/app/~~t_t-ZausPkFT7OqxtUcQ4A==/com.example.flutter_rust_test-6UUUaPpOY4r5WiJSCaJqNg==/lib/arm64/librust_lib_flutter_rust_test.so
#12 pc 00000000000a7df4 /data/app/~~t_t-ZausPkFT7OqxtUcQ4A==/com.example.flutter_rust_test-6UUUaPpOY4r5WiJSCaJqNg==/lib/arm64/librust_lib_flutter_rust_test.so
#13 pc 00000000000b4880 /data/app/~~t_t-ZausPkFT7OqxtUcQ4A==/com.example.flutter_rust_test-6UUUaPpOY4r5WiJSCaJqNg==/lib/arm64/librust_lib_flutter_rust_test.so
#14 pc 00000000000b324c /data/app/~~t_t-ZausPkFT7OqxtUcQ4A==/com.example.flutter_rust_test-6UUUaPpOY4r5WiJSCaJqNg==/lib/arm64/librust_lib_flutter_rust_test.so
#15 pc 0000000000196d8c /data/app/~~t_t-ZausPkFT7OqxtUcQ4A==/com.example.flutter_rust_test-6UUUaPpOY4r5WiJSCaJqNg==/lib/arm64/librust_lib_flutter_rust_test.so
#16 pc 00000000001731a4 /data/app/~~t_t-ZausPkFT7OqxtUcQ4A==/com.example.flutter_rust_test-6UUUaPpOY4r5WiJSCaJqNg==/lib/arm64/librust_lib_flutter_rust_test.so
#17 pc 00000000001799fc /data/app/~~t_t-ZausPkFT7OqxtUcQ4A==/com.example.flutter_rust_test-6UUUaPpOY4r5WiJSCaJqNg==/lib/arm64/librust_lib_flutter_rust_test.so
#18 pc 0000000000179400 /data/app/~~t_t-ZausPkFT7OqxtUcQ4A==/com.example.flutter_rust_test-6UUUaPpOY4r5WiJSCaJqNg==/lib/arm64/librust_lib_flutter_rust_test.so
#19 pc 000000000002de10 /system/lib64/libaaudio_internal.so (aaudio::AudioStream::maybeCallDataCallback(void*, int)+192) (BuildId: 0cf8e802c015dcff49dfa99f8e9b4983)
#20 pc 00000000000321d0 /system/lib64/libaaudio_internal.so (aaudio::AudioStreamLegacy::callDataCallbackFrames(unsigned char*, int)+304) (BuildId: 0cf8e802c015dcff49dfa99f8e9b4983)
#21 pc 00000000000310cc /system/lib64/libaaudio_internal.so (aaudio::AudioStreamLegacy::onMoreData(android::AudioTrack::Buffer const&)+636) (BuildId: 0cf8e802c015dcff49dfa99f8e9b4983)
#22 pc 00000000000a84f0 /system/lib64/libaudioclient.so (android::AudioTrack::processAudioBuffer()+2832) (BuildId: 3cbde63a227ac7432c8e29bb4c1fa488)
#23 pc 00000000000a76e0 /system/lib64/libaudioclient.so (android::AudioTrack::AudioTrackThread::threadLoop()+272) (BuildId: 3cbde63a227ac7432c8e29bb4c1fa488)
#24 pc 00000000000115d4 /system/lib64/libutils.so (android::Thread::_threadLoop(void*)+244) (BuildId: c07f08c7e5a964a8f8c6bc5c820fb795)
#25 pc 00000000000edf3c /system/lib64/libandroid_runtime.so (android::AndroidRuntime::javaThreadShell(void*)+140) (BuildId: 07fe69a1909e86b0aa90b83a17bd2e07)
#26 pc 000000000006efbc /apex/com.android.runtime/lib64/bionic/libc.so (__pthread_start(void*)+204) (BuildId: 1d36f8ae6e0af6158793abea7d4f4f2b)
#27 pc 0000000000060d60 /apex/com.android.runtime/lib64/bionic/libc.so (__start_thread+64) (BuildId: 1d36f8ae6e0af6158793abea7d4f4f2b)
Lost connection to device.
Exited.
I honestly don't have any clue why this is happening. I did notice that the error only happened if the chunk.commit_all() was there, but that is a critical line because it frees the ring buffer back up. Without it, the error goes away but no sound is played. I don't think this is an issue with the ring buffer crate, because it seems to work multiple times before the whole program crashes. I took a look at this issue and this one as well, with no luck. I also tried this on an iOS device and it worked perfectly, so I'm not entirely sure why this isn't working on Android. I wish I had more information to share, but I honestly don't know anything else. Any ideas?
Edit: I found that this issue stemmed from this program not taking into account fluctuations of the length of the input data. At certain times the program would try and copy some empty data into the output buffer, and I think this caused the crash. This leads me to try and diagnose the next issue - if I were to print data.len() inside the data callback of input_stream:
debug!("input_config: len of input data is {}", data.len());
I get this output:
D/rust_lib_flutter_rust..(25504): input_config: len of input data is 960
D/rust_lib_flutter_rust..(25504): input_config: len of input data is 960
D/rust_lib_flutter_rust..(25504): input_config: len of input data is 960
D/rust_lib_flutter_rust..(25504): input_config: len of input data is 640
D/rust_lib_flutter_rust..(25504): input_config: len of input data is 320
D/rust_lib_flutter_rust..(25504): input_config: len of input data is 960
D/rust_lib_flutter_rust..(25504): input_config: len of input data is 960
D/rust_lib_flutter_rust..(25504): input_config: len of input data is 960
D/rust_lib_flutter_rust..(25504): input_config: len of input data is 896
D/rust_lib_flutter_rust..(25504): input_config: len of input data is 64
notice that it fluctuates a lot but tends to hang around 960. I did ask a similar question about this in another comment a few months back, but what really confuses me here in particular is the fact that I set the input config's buffer size to be fixed to 480:
const BUFFER_SIZE: u32 = 480;
let input_config = cpal::StreamConfig {
channels: 1,
sample_rate: cpal::SampleRate(48000),
buffer_size: cpal::BufferSize::Fixed(BUFFER_SIZE as u32)
};
yet, it seems like this isn't being adhered to at all in the input. is this behavior something that I would just have to make my program take into account and work around, or is something wrong here?
I've just encountered this problem too. Leading to some glitchiness in the audio.
I found the relevant part in ndk::audio which is what cpal calls into.
pub fn buffer_capacity_in_frames(self, num_frames: i32) -> Self
"Set the requested buffer capacity in frames. The final AAudioStream capacity may differ, but will probably be at least this big." (source)
edit: found solution see below
~so perhaps the only way on android is to get the buffer size dynamically by looking at data.len() each callback instead of relying on a static value..~
~A bit of a nuisance with the inconsistency, all other platforms I compile for the buffer size can be expected to be actually fixed but whatchugonnado I reckon this is an android implementation thing and perhaps unavoidable~
some hours later & I found the answer to this android has a low latency mode but it needs to explicitly requested
ndk::audio has the option to do so in their builder but cpal does not expose it in the api as far as i understand
I added AudioStreamBuilder::performance_mode(AudioPerformanceMode::LowLatency) during input/output stream creation - see: https://github.com/amomentunfolding/cpal/commit/a1268d5eb1ace149552b8dc8887291b10036607d
and now the buffer size does not fluctuate over time it is still not what I requested though
requested buffer size: 256
len input data is 96
len output data is 512
but i am quite pleased with this, i do not get any more glitches or wonkyness. though hopefully i can reduce the output latency eventually too we shall see
