auv3 + midi out doesn't pass auvaltool on macos 12

[modosc]

i've tested this on intel and arm64 but only on macos12. ~~i know for a fact this worked in april of this year but then i was on macos 11 and an intel mac.~~ (edit: j/k, i didn't have midi out enabled then, i can't verify if this ever worked).

to reproduce:

1. make sure juce is on develop and projucer is built from it.
2. open/export the AUv3SynthPlugin example
3. make sure plugin AU main type is set to kAudioUnit_MusicDevice
4. make sure plugin AU is sandbox safe is enabled
5. make sure Use App Sandbox is enabled
6. make sure there's a valid value in Development Team ID
7. make sure plugin is a synth and plugin midi input are checked.

now build everything and open the standalone executable once. after this, you should be able to successfully run auvaltool (replacing the last two values with whatever projucer filled them in with):

$ auvaltool -v aumu Noyv Manu
...
Test MIDI
  PASS

* * PASS
--------------------------------------------------
AU VALIDATION SUCCEEDED.
--------------------------------------------------

great. now go back to projucer and check plugin midi output. save, re-export, rebuild, and try auvaltool again:

$ auvaltool -v aumu Noyv Manu
...
Test MIDI
ERROR: -66745 IN CALL AudioUnitRender

* * FAIL
--------------------------------------------------
AU VALIDATION FAILED: CORRECT THE ERRORS ABOVE.
--------------------------------------------------

i couldn't debug with auvaltool on arm64 but i managed to get a stack trace on an intel mac:

[modosc]

adding my updates from the forum post:

1. if i build an au instead of an auv3 then the result does pass auvaltool
2. i found a report of almost the exact same bug on ios: https://twitter.com/bram_bos/status/1444214951640449028?s=20

[izzyreal]

For anyone interested in targeting systems that do support MIDI output from an instrument while also targeting an AUv3 with MIDI output disabled, I did the following hack:

+++ b/modules/juce_audio_plugin_client/AU/juce_AUv3_Wrapper.mm
@@ -1625,7 +1625,7 @@ AUAudioUnitStatus renderCallback (AudioUnitRenderActionFlags* actionFlags, const
 
             // send MIDI
            #if JucePlugin_ProducesMidiOutput && JUCE_AUV3_MIDI_OUTPUT_SUPPORTED
-            if (@available (macOS 10.13, iOS 11.0, *))
+            if (@available (macOS 10.13, iOS 11.0, *) && getAudioProcessor().producesMidi())^M
             {
                 if (auto midiOut = midiOutputEventBlock)
                     for (const auto metadata : midiMessages)

And then in your AudioProcessor you can do

bool YourAudioProcessor::producesMidi() const
{
  if (wrapperType == wrapperType_AudioUnitv3) return false;
#if JucePlugin_ProducesMidiOutput
  return true;
#else
  return false;
#endif
}

Having optional AUv3-incompatible bus layouts can be done without any JUCE hacks:

YourAudioProcessor::YourAudioProcessor()
: AudioProcessor (juce::PluginHostType::jucePlugInClientCurrentWrapperType == juce::AudioProcessor::wrapperType_AudioUnitv3 ?
                  BusesProperties()
                  .withOutput("STEREO OUT", juce::AudioChannelSet::stereo(), true)
                  :
                  BusesProperties()
                  .withInput("RECORD IN",  juce::AudioChannelSet::stereo(), true)
                  .withOutput("STEREO OUT", juce::AudioChannelSet::stereo(), true)
                  )