WIN10-ASIO 0xc0000005 Access Violation (DEP) When Both Streams Built

[keeripes]

I see there are some other issues similar to this, and I have been banging my head against the wall for a couple of days now with this issue which I will outline as succinctly as possible.

The Error

I am getting the same behavior on two machines with different audio interfaces:

• when only an input stream is initialized I successfully get inbound audio from the audio interface.
• when both input and output streams are initialized I get the access violation shown below
• when only an output stream is initialized I get output audio on one machine/interface pair and the exception on the other.

The error is the same - after the stream starts I get the error:

error: process didn't exit successfully: '...' (exit code: 0xc0000005, STATUS_ACCESS_VIOLATION)

If I step either through with a debugger I get a bit more information:

Exception: Exception 0xc0000005 encountered at address 0x000060: User-mode data execution prevention (DEP) violation at location 0x00000060

(Not sure if this is a debugger artifact or not... I just wanted to put it here)

This access violation exception happens randomly while stepping through in debug, leading me to think it might be caused by the ASIO bufferSwitch callback - maybe the first time it gets called on the output buffers.

I am getting this same behavior on two different machines with two different interfaces (in both debug and release builds):

• Machine 1 - Windows 10 (64-bit) VM on QEMU w/ USB passthrough with a MOTU UltraLiteMk3.

• Machine 2 - Windows 10 (64-bit) Bare metal with a Digigram VX882 PCIe card.

I should note here that both interfaces work flawlessly with REAPER on their parent machines. I do not suspect a driver issue. Likely a me issue.

The test case - feedback.rs

In its current form, the feedback.rs example doesn't work out of the box with ASIO (on my machines), so in an effort to troubleshoot I cloned the cpal repo and started poking around. After looking at the ASIO implementation in cpal, I saw two things that jumped out given the exception is potentially due to Data Execution Prevention. In my feeble mind I start thinking that something has moved on the rust side, invalidating the pointers held either by the driver (audio buffers) or the ASIO callbacks on the rust side.

The two things I see as possible pinch points (though they appear 100% correct) are:

• the audio buffers are split apart into different Options after being initialized via ASIOCreateBuffers in (Driver::create_streams())
• when the second opposing direction stream is created, new buffers and ASIO callback pointers are created (Driver::create_buffers()).

So as a test I added a few hacky functions to allow for both streams and callbacks to be built at the same time and to my surprise: IT WORKED on the bare metal machine. I am successfully feeding back audio.

The VM is still kicking out an access violation if any output buffers are initialized. Might be due to Ballooning memory or a myriad of things - so I am focusing on the bare metal machine.

Hacky workaround

I tried to modify as little code as possible. Important to note here is that ALL Bufferinfos are stored in the asio_stream.input Option and that's why you see the added functions referencing the input member of the stream in the output stream

The main.rs:

use clap::ArgMatches;
use crate::errors::AudioPlatformError;
use cpal::traits::{HostTrait, DeviceTrait, StreamTrait};
use ringbuf::RingBuffer;
use dasp::{Sample, Signal};
use dasp::sample::I24;
use dasp::ring_buffer::SliceMut;

pub fn asio_scratch_testing(matches: &ArgMatches) -> Result<(), AudioPlatformError> {
    println!("Let's get started!");

    // Baremetal test case
    // let device_name = "ASIO for VX882HR";
    // VM test case
    let device_name = "MOTU Audio ASIO";
    // Loading block
    let host = cpal::host_from_id(cpal::HostId::Asio)?;
    let channels = 14;
    let sample_rate = 48000;
    let mut devices = host.input_devices()?;
    let device = devices.find(|x| x.name().map(|y| y.as_str() == device_name).unwrap_or(false)).expect("Couldn't find device!");

    
    let length: usize = 8 * 48000 * 2;
    // Setup stream config
    let config: cpal::StreamConfig = cpal::StreamConfig {
        channels,
        sample_rate: cpal::SampleRate(sample_rate),
        buffer_size: cpal::BufferSize::Default
    };

    let err_fn = move |err| {
        eprintln!("{:?}", err);
    };

    let output_data_fn = move |data: &mut [i32], c: &cpal::OutputCallbackInfo| {
        let len = data.len();
        for offset in (0..len).step_by(28) {
            for (i, o) in (0..14).zip(14..28) {
                data[o + offset] = data[i + offset];
            }

        }
    };

    let output_stream = device.build_output_stream(&config, output_data_fn, err_fn)?;
    output_stream.play()?;

    // give it enough time to fail
    std::thread::sleep(std::time::Duration::from_secs(15));
    Ok(())
}

In src/host/asio/stream.rs:

impl Device {
    // ...
    // commented out original build_output_streams_raw() and replaced with this
    /// A quick outline of the "test" changes - function name no longer accurately represents
    /// what this does - I am "overwriting" this method because it is called by the trait
    /// and it has most of what I need.
    /// added `create_bidirectional_stream` method
    /// The interleaved callback buffer now contains both input and output sample data
    /// where I is an input sample and O is an output sample the interleaved buffer now
    /// looks like this:
    /// [I, I, I, I, I,..., I, O, O, O, O, ..., O]
    /// so this has to be dealt with in the "user" callback
    ///
    /// So now, 
    /// Input samples are put in the interleaved buffer - the user callback is called
    /// then the output samples are pulled out of the interleaved buffer.
    pub fn build_output_stream_raw<D, E>(
        &self,
        config: &StreamConfig,
        sample_format: SampleFormat,
        mut data_callback: D,
        _error_callback: E,
    ) -> Result<Stream, BuildStreamError>
        where
            D: FnMut(&mut Data, &OutputCallbackInfo) + Send + 'static,
            E: FnMut(StreamError) + Send + 'static,
    {
        let stream_type = self.driver.output_data_type().map_err(build_stream_err)?;

        // Ensure that the desired sample type is supported.
        let expected_sample_format = super::device::convert_data_type(&stream_type)
            .ok_or(BuildStreamError::StreamConfigNotSupported)?;
        if sample_format != expected_sample_format {
            return Err(BuildStreamError::StreamConfigNotSupported);
        }

        let num_channels = config.channels.clone() * 2;
        
        // Using a different stream build method
        let buffer_size = self.create_bidirectional_stream(config, sample_format)?;
        let cpal_num_samples = buffer_size * num_channels as usize;

        // Create buffers depending on data type.
        let len_bytes = cpal_num_samples * sample_format.sample_size();
        let mut interleaved = vec![0u8; len_bytes];
        let mut silence_asio_buffer = SilenceAsioBuffer::default();

        let stream_playing = Arc::new(AtomicBool::new(false));
        let playing = Arc::clone(&stream_playing);
        let asio_streams = self.asio_streams.clone();

        let config = config.clone();
        let callback_id = self.driver.add_callback(move |callback_info| unsafe {
            // If not playing, return early.
            if !playing.load(Ordering::SeqCst) {
                return;
            }

            // There is 0% chance of lock contention the host only locks when recreating streams.
            let stream_lock = asio_streams.lock();
            let ref asio_stream = match stream_lock.input {
                Some(ref asio_stream) => asio_stream,
                None => return,
            };

            // Silence the ASIO buffer that is about to be used.
            //
            // This checks if any other callbacks have already silenced the buffer associated with
            // the current `buffer_index`.
            //
            // If not, we will silence it and set the opposite buffer half to unsilenced.
            let silence = match callback_info.buffer_index {
                0 if !silence_asio_buffer.first => {
                    silence_asio_buffer.first = true;
                    silence_asio_buffer.second = false;
                    true
                }
                0 => false,
                1 if !silence_asio_buffer.second => {
                    silence_asio_buffer.second = true;
                    silence_asio_buffer.first = false;
                    true
                }
                1 => false,
                _ => unreachable!("ASIO uses a double-buffer so there should only be 2"),
            };

            /// 1. Render the given callback to the given buffer of interleaved samples.
            /// 2. If required, silence the ASIO buffer.
            /// 3. Finally, write the interleaved data to the non-interleaved ASIO buffer,
            ///    performing endianness conversions as necessary.
            unsafe fn process_output_callback<A, B, D, F>(
                data_callback: &mut D,
                interleaved: &mut [u8],
                silence_asio_buffer: bool,
                asio_stream: &sys::AsioStream,
                asio_info: &sys::CallbackInfo,
                sample_rate: crate::SampleRate,
                to_endianness: F,
            ) where
                A: Sample,
                B: AsioSample,
                D: FnMut(&mut Data, &OutputCallbackInfo) + Send + 'static,
                F: Fn(B) -> B,
            {
                // 1. Render interleaved buffer from callback.
                let interleaved: &mut [A] = cast_slice_mut(interleaved);
                let buffer_index = asio_info.buffer_index as usize;
                let callback = system_time_to_stream_instant(asio_info.system_time);
                let n_frames = asio_stream.buffer_size as usize;
                let n_channels = interleaved.len() / n_frames;

                // Put the input samples into the interleaved buffer
                for ch_ix in 0..n_channels / 2 {
                    let asio_channel = asio_channel_slice::<B>(asio_stream, buffer_index, ch_ix);
                    for (frame, s_asio) in interleaved.chunks_mut(n_channels).zip(asio_channel) {
                        frame[ch_ix] = (*s_asio).to_cpal_sample::<A>();
                    }
                }

                let data = interleaved.as_mut_ptr() as *mut ();
                let len = interleaved.len();
                let mut data = Data::from_parts(data, len, A::FORMAT);

                let delay = frames_to_duration(n_frames, sample_rate);
                let playback = callback
                    .add(delay)
                    .expect("`playback` occurs beyond representation supported by `StreamInstant`");
                let timestamp = crate::OutputStreamTimestamp { callback, playback };
                let info = OutputCallbackInfo { timestamp };
                data_callback(&mut data, &info);

                // 2. Silence ASIO channels if necessary. Commented out for simplicity

                // if silence_asio_buffer {
                //     for ch_ix in (n_channels/2)..n_channels {
                //         let asio_channel =
                //             asio_channel_slice_mut::<B>(asio_stream, buffer_index, ch_ix);
                //         asio_channel
                //             .iter_mut()
                //             .for_each(|s| *s = to_endianness(B::SILENCE));
                //     }
                // }

                // Yyyyoink the output samples from the interleaved buffer and write them into
                // the ASIO buffers
                for ch_ix in (n_channels / 2)..n_channels {
                    let asio_channel =
                        asio_channel_slice_mut::<B>(asio_stream, buffer_index, ch_ix);
                    for (frame, s_asio) in interleaved.chunks(n_channels).zip(asio_channel) {
                        *s_asio = to_endianness(B::from_cpal_sample(&frame[ch_ix]));
                    }
                }
            }
        }
                                                   // ... and the rest of the function is identical from here.
        );
        // ...
    }
    //...

    /// The primary substitution method to ensure that both callbacks, output and input streams are created
    /// at the same time.  The current issue is a STATUS_ACCESS_VIOLATION will occur when
    /// both streams are created separately. The exception is a Data Execution Prevention error
    /// and I did notice that the callbacks would work normally when only a single stream was
    /// running.  
    fn create_bidirectional_stream(
        &self,
        config: &StreamConfig,
        sample_format: SampleFormat,
    ) -> Result<usize, BuildStreamError> {
        match self.default_input_config() {
            Ok(f) => {
                let num_asio_channels = f.channels;
                check_config(&self.driver, config, sample_format, num_asio_channels)
            }
            Err(_) => Err(BuildStreamError::StreamConfigNotSupported),
        }?;
        let num_channels = config.channels as usize;
        let ref mut streams = *self.asio_streams.lock();

        let buffer_size = match config.buffer_size {
            BufferSize::Fixed(v) => Some(v as i32),
            BufferSize::Default => None,
        };

        // Either create a stream if thers none or had back the
        // size of the current one.
        // This will be None 100% of the time.
        match streams.input {
            Some(ref input) => Ok(input.buffer_size as usize),
            None => {
                self.driver
                    .prepare_both_streams(num_channels, buffer_size)
                    .map(|new_streams| {
                        let bs = match new_streams.input {
                            Some(ref inp) => inp.buffer_size as usize,
                            None => unreachable!(),
                        };
                        *streams = new_streams;
                        bs
                    })
                    .map_err(|ref e| {
                        println!("Error preparing stream: {}", e);
                        BuildStreamError::DeviceNotAvailable
                    })
            }
        }
    }
    // ...
}

In asio-sys/src/bindings/mod.rs:

impl Driver {
    
    // ...
    /// This is a quick hack to be used in the custom `create_bidirectional_streams()` which will
    /// be used instead of `get_or_build_output_stream()` in the `build_output_stream_raw()`function.
    /// This function simply makes sure the same amount of input and output buffers (and streams)
    /// are created simultaneously.  This has some obvious limitations in that number of inputs
    /// will always be forced to be equal to number of outputs.
    pub fn prepare_both_streams(
        &self,
        num_channels: usize,
        buffer_size: Option<i32>,
    ) -> Result<AsioStreams, AsioError> {
        let input_buffer_infos = prepare_buffer_infos(true, num_channels);
        let output_buffer_infos = prepare_buffer_infos(false, num_channels);
        self.unified_create_streams(input_buffer_infos, output_buffer_infos, buffer_size)
    }
    
    // ...

    /// A modified copy of create_streams() but does not split the buffers,
    /// meaning both input and output buffers now reside on the same "direction", and
    /// since only the output callbacks deliver a mutable reference to the ASIO buffers
    /// to the "user callback" that ""direction will be output - input is always None
    /// Since this is a quick hack and function is only called from `prepare_both_streams`
    /// I TRIED removing the extra match conditions for clarity but that broke everything???
    /// put them back and everything started working again.
    fn unified_create_streams(
        &self,
        mut input_buffer_infos: Vec<AsioBufferInfo>,
        mut output_buffer_infos: Vec<AsioBufferInfo>,
        buffer_size: Option<i32>,
    ) -> Result<AsioStreams, AsioError> {
        let (input, output) = match (
            input_buffer_infos.is_empty(),
            output_buffer_infos.is_empty(),
        ) {
            // Both stream exist.
            (false, false) => {
                // Create one continuous slice of buffers.
                // println!("create streams called with both streams existing");
                let split_point = input_buffer_infos.len();
                let mut all_buffer_infos = input_buffer_infos;
                all_buffer_infos.append(&mut output_buffer_infos);
                // Create the buffers. On success, buffers are no longer split.
                let buffer_size = self.create_buffers(&mut all_buffer_infos, buffer_size)?;
                // let output_buffer_infos = all_buffer_infos.split_off(split_point);
                // let input_buffer_infos = all_buffer_infos;
                let input = Some(AsioStream {
                    buffer_infos: all_buffer_infos,
                    buffer_size,
                });
                let output = None;
                (input, output)
            }
            // Just input
            (false, true) => {
                // println!("create streams called with output stream existing (input empty)");
                let buffer_size = self.create_buffers(&mut input_buffer_infos, buffer_size)?;
                let input = Some(AsioStream {
                    buffer_infos: input_buffer_infos,
                    buffer_size,
                });
                let output = None;
                (input, output)
            }
            // Just output
            (true, false) => {
                // println!("create streams called with input stream existing (output empty)");
                let buffer_size = self.create_buffers(&mut output_buffer_infos, buffer_size)?;
                let input = None;
                let output = Some(AsioStream {
                    buffer_infos: output_buffer_infos,
                    buffer_size,
                });
                (input, output)
            }
            // Impossible
            _ => unreachable!("Trying to create streams without preparing"),
        };
        Ok(AsioStreams { input, output })
    }
    
}

Moving into WTF territory

So once I had everything working on the bare metal machine, I figured I would clean up any extranneous code to make this post a little bit cleaner. However, when I remove the unused match conditions in unified_create_streams (since the parent function forces the (false, false) pattern) the audio no longer works and I get and immediate STATUS_ACCESS_VIOLATION on the next run of the program. I put the code back where it belonged and VOILA! everything was working again.

I can only surmise I am doing something severely wrong.

Does anybody see anything obvious? Could it be compiler optimizations moving stuff around? This whole process has felt like balancing on one foot in a canoe in the middle of the ocean.

Why post this issue

Do I believe this is a greater cpal issue directly? Not really, however I hope some of the expertise surrounding this community might see a pattern I am missing and perhaps enlighten myself and the others facing these nebulous ASIO issues when using this crate as to what we could be doing to cause this issue.

Or can we blame Billy Gates? Is something going on with the shared memory and pointers between the rust binary/bindings and the often hastily vendor-developed ASIO drivers?

Any help is good help, even if it means outing myself as a complete dunce.

Thanks!

[Ralith]

I do not suspect a driver issue. Likely a me issue.

FWIW, if you're able to reproduce this without writing unsafe in your code, then this is a CPAL or driver issue, full stop.

[keeripes]

I can confirm that no unsafes were added, only copy-paste-modify the existing cpal code in the /host/asio area of the repo. [edited for grammar]

I will say that the CPAL implementation seems to be making the issue better, and I'll quickly explain why.

To keep it as short as possible I left out the fact that I have also done testing using the bindings in asio-sys directly and I can say that when using the bindings outside of CPAL I am able to recreate this issue with only a minor (but hopefully telling) difference - Output streams never work. Input streams always work.

I will be the first to admit my understanding of Rust's closure stack/heap/memory access model is right around 0% but I am getting the sense that CPAL + bindings is working slightly better because of the way the ASIO callbacks are moved from within the scope of the parent struct which in itself contains the actual "driver". I realize they are Arc::clone() mutexes that get moved into the closure, but I have this feeling that some part of the code layout is causing Rust to compile things in a way that keeps everything accessible (maybe same page on the heap?). For instance the lazy_static Callback variable is in the same file and referenced in the same object and maybe that helps keep the callbacks accessible? If that sounds dumb, then I agree, it probably is, haha.

The DEP exception is giving me the impression that the ASIO driver is trying to jump to a callback/closure that has either been moved or potentially dropped.

When messing with the asio-sys bindings directly I recreated many aspects of the approach CPAL uses, and the only difference is the output streams sometimes work with CPAL + bindings and never with the bindings directly. With the direct bindings approach, if I put a BufferCallbackInfo with a isInput field of 0 I am guaranteed to get an exception.

I realize that isn't a great description, but I don't want to cloud the issue too much.

I just hope my blubbering makes someone have an 'aha!' moment.

[est31]

#775 should have fixed this, if not, please write!