runner-images icon indicating copy to clipboard operation
runner-images copied to clipboard

Core Audio taking 3 minutes to open audio device

Open miketoon opened this issue 1 year ago • 14 comments

Description

When starting an audio device with CoreAudio it takes 3 minutes to return when calling:

AudioDeviceStart(   AudioObjectID                   inDevice,
                    AudioDeviceIOProcID __nullable  inProcID)                                                   __OSX_AVAILABLE_STARTING(__MAC_10_0, __IPHONE_2_0);

We are trying to upgrade from macosx12 but are running into this issue with our tests where we need to repeatedly start an audio device. This is causing a massive delay to the tests which is unworkable.

Platforms affected

  • [ ] Azure DevOps
  • [ ] GitHub Actions - Standard Runners
  • [ ] GitHub Actions - Larger Runners

Runner images affected

  • [ ] Ubuntu 20.04
  • [ ] Ubuntu 22.04
  • [ ] Ubuntu 24.04
  • [ ] macOS 12
  • [X] macOS 13
  • [ ] macOS 13 Arm64
  • [X] macOS 14
  • [ ] macOS 14 Arm64
  • [X] macOS 15
  • [ ] macOS 15 Arm64
  • [ ] Windows Server 2019
  • [ ] Windows Server 2022

Image version and build link

macos13: Current image version: '20241023.237' macos 15: Current image version: '20241022.244'

Is it regression?

Yes macosx12

Expected behavior

Audio device is started in a few seconds or less

Actual behavior

Audio device doesn't start for 3 minutes hanging the application

Repro steps

call to

AudioDeviceStart(   AudioObjectID                   inDevice,
                    AudioDeviceIOProcID __nullable  inProcID)                                                   __OSX_AVAILABLE_STARTING(__MAC_10_0, __IPHONE_2_0);

miketoon avatar Nov 05 '24 10:11 miketoon

Hi @miketoon We will look into the issue and keep you posted with updates.

susmitamane avatar Nov 05 '24 11:11 susmitamane

@miketoon , I can't seem to reproduce this. I tried using an application that utilizes AudioDeviceStart. Can you provide me with a way to reproduce this?

NorseGaud avatar Nov 18 '24 16:11 NorseGaud

@NorseGaud Please see example app in this zip: CoreAudioTest.zip

miketoon avatar Nov 19 '24 12:11 miketoon

Here is what I'm seeing on the first run.

Screenshot 2024-11-19 at 8 12 37 AM

Second run, without clicking "Allow", sits for a long time and then finally fails at:

Screenshot 2024-11-19 at 8 16 29 AM

Once I allow the Microphone, I see things work fast for the core audio logs:

Screenshot 2024-11-19 at 8 18 00 AM

I am not using a GitHub runner Mac though, so this is a pure VM. @susmitamane , check with the team and see if the Microphone permissions are the issue by manually testing this in Github's environment/VMs.

NorseGaud avatar Nov 19 '24 14:11 NorseGaud

The timing matches up with what I am seeing when on a runner. This is strange because we are attempting to start the device without any audio input so I am not sure why access is even being requested.

miketoon avatar Nov 19 '24 14:11 miketoon

It may be a consequence of Apple's design :( I think there is a way around this. Can you clarify what binaries/commands you run to build/test? There is a way to preapprove certain binaries for certain permissions in the Github runner images but the team needs to know exactly what you're running that does the build.

NorseGaud avatar Nov 19 '24 14:11 NorseGaud

We are building our binary using XCode Our tests are ran from a Rake script which calls Open3.popen2e(<ARGS>) to run the binary.

miketoon avatar Nov 19 '24 14:11 miketoon

Sorry, so the issue isn't with building but running your binary? Let's see what the Github team recommends here. It could be the permissions for rake weren't added to the VM when it was created, so try running the binary without it and see if it's still a problem.

Is your binary and rake script something you could share privately with us so we can test again? Maybe you can create a tiny version of the script with just the necessities?

NorseGaud avatar Nov 19 '24 18:11 NorseGaud

The same issue happens when running the binary directly on the runner. I think this is what you've already seen when running the example app and the popup appears.

Output running directly on runner:

Generating script.
Script contents:
./Builds/MacOSX/build/Release/FIngerprintTests
========================== Starting Command Output ===========================
/bin/bash --noprofile --norc /Users/runner/work/_temp/07c3b6fc-9442-43e4-a37f-2e88fb8d5ca8.sh
Tracktion DM: init 0 seconds
Tracktion DM: scan end 0 seconds
Tracktion DM: pick end 0 seconds
Tracktion DM: init default start 0 seconds
Tracktion DM: init default AudioDeviceSetup end 0 seconds
Tracktion DM: Insert default device names 0 seconds
Tracktion DM: type CoreAudio 0 seconds
Tracktion DM: output to test Null Audio Device 0 seconds
Tracktion DM: insertDefaultDeviceNames end 0 seconds
Tracktion DM: setAudioDeviceSetup start 0 seconds
Tracktion DM: stop Device start 0 seconds
Tracktion DM: setAudioDeviceSetup end 0 seconds
Tracktion DM: type CoreAudio 0 seconds
Tracktion DM: needsNewDevice start 0 seconds
Tracktion DM: deleteCurrentDevice start 0 seconds
Tracktion DM: deleteCurrentDevice end 0 seconds
Tracktion DM: type CoreAudio 0 seconds
Tracktion DM: updateSetupChannels start 0 seconds
Tracktion DM: updateSetupChannels end 0 seconds
Tracktion DM: currentAudioDevice->open start 0 seconds
Tracktion DM: 0 : 2 0 seconds
Tracktion DM: currentAudioDevice->open end 0 seconds
Tracktion DM: currentAudioDevice->start start 0 seconds
Tracktion CA: pre lock  0 seconds
Tracktion CA: post lock 0 seconds
Tracktion DM: currentAudioDevice->start end 180 seconds
Tracktion DM:  180 seconds
Tracktion DM: setAudioDeviceSetup end 180 seconds
Tracktion DM: init default end 180 seconds

Finishing: Run FIngerprintTests (Mac)

miketoon avatar Nov 20 '24 10:11 miketoon

@NorseGaud any updates from the GitHub team? Or suggestions to how the required permissions can be given for our binary?

miketoon avatar Nov 26 '24 10:11 miketoon

@NorseGaud any update on this?

miketoon avatar Dec 03 '24 17:12 miketoon

Hey @miketoon , I actually don't work for Github, I was just helping out. You'll need to wait for their team to get back to you. I've let them know what I recommend to test and fix this.

NorseGaud avatar Dec 03 '24 17:12 NorseGaud

Hi @miketoon , Thank you for your patience. I am currently investigating the issue and it is in progress. Please allow us some more time to delve deeper. So far, I have reproduced the steps locally using Xcode and imported the files you shared, but I am not receiving any prompts for microphone permissions. Please see below snip. Will come back with some updates later. Thank you.

Screenshot 2024-11-27 at 1 02 51 PM

aartis17 avatar Dec 09 '24 13:12 aartis17

@aartis17 Is this running on the runner image? I don't get the prompt on my dev machine running locally. Have you made any further progress?

miketoon avatar Dec 17 '24 09:12 miketoon

@miketoon No, It was from Xcode tool which was running on my local machine. We have investigated the issue, we couldn't able to reproduce it, as we cannot see any Yaml files there in zip file, can you please share the path if the files are present. Also, could you please share complete repro steps, Github workflow (.yaml files) and please share public Github repository link if any. Thanks.

aartis17 avatar Dec 26 '24 06:12 aartis17

I am also facing some issue in playing audio on agent with macos-14 image https://github.com/actions/runner-images/issues/11251

sanket492 avatar Jan 08 '25 09:01 sanket492

@aartis17 fingerprinttests_steps_yaml.zip Running locally i do not get the same issue also, only when it is run on the runner image. The repro steps are simple - build that application and run it on the runner image. The app that is built is an audio application, but it does not need audio input access. As you see locally and I see locally this is not always requested when the app is ran. My suspicion was that the hardware configuration of the runner image needs to request access for some reason purely so that we can output audio. Could that be possible? Is there a way to give this permission to apps programmatically? Or is there a way to run this console application in a context where this permission would already be granted? Or can the microphone be disabled on the runner image?

miketoon avatar Jan 14 '25 10:01 miketoon

managed to prevent this by running:

sqlite3 $HOME/Library/Application\ Support/com.apple.TCC/TCC.db "INSERT OR IGNORE INTO access VALUES ('kTCCServiceMicrophone','/usr/local/opt/runner/provisioner/provisioner',1,2,4,1,NULL,NULL,0,'UNUSED',NULL,0,1687786159,NULL,NULL,'UNUSED',1687786159);"

as suggested from https://github.com/actions/runner-images/issues/9330

@aartis17 Can this be incorporated by default into the runner images?

miketoon avatar Jan 15 '25 12:01 miketoon

Looks like it needs to be added to https://github.com/actions/runner-images/blob/main/images/macos/scripts/build/configure-tccdb-macos.sh

NorseGaud avatar Jan 15 '25 16:01 NorseGaud

Yes @miketoon @NorseGaud, We can accomplish this by updating the tcc db to include the necessary values for granting microphone permissions. I'll prepare the pull request and keep you updated on the progress. Thanks.

aartis17 avatar Jan 16 '25 06:01 aartis17

@aartis17 brilliant, thank you!

miketoon avatar Jan 16 '25 09:01 miketoon

Hi @miketoon, thanks for your patience. We've deployed the new images along with the changes in the TCC database. Could you please verify if the issue has been resolved now? Thanks.

aartis17 avatar Jan 30 '25 07:01 aartis17

@aartis17 looks to be fixed on the latest macos14 runner. Thank you!

miketoon avatar Jan 30 '25 16:01 miketoon

@miketoon Thanks for your response. Since the solution seems to be working well, are we good to close this issue now?

aartis17 avatar Jan 31 '25 07:01 aartis17