Audio visualizer

[BradyBrenot]

Design notes

• It's an Effect so it can overlay with other Effects. Can safely leave "no sound" as "transparent"
• Try to see what different things look like and possibly use in combination, possibly with options
  • Onset ("beat") detection
    • Could cause a pulse or a wave that expands, or just an instantaneous effect might be better
    • Could be modified by frequency band intensity (FFT) detection (below)
    • Localize it with dematrixing
  • Frequency band intensity
    • Colors the light, or changes intensity, or both. Common strategy in visualizers. Commonly the only strategy.
  • Dematrixing
    • Try to make a 2D location from the sound instead of just 1D. Try to get some spatialization into it.

Expanding wave

I dunno

• Griddy approach
  • Like fluid sim? Fluid sim is a bad idea, I don't actually want that effect, as cool as it might sound it won't be very reactive. Never mind how much of a pain it is.
• Eventy approach
  • Every hit makes a 'pulse', save as time/intensity/color and then calculate contribution of all pulses to specific light every frame.
    • Problem if too many simultaneous pulses.
      • Limit # of pulses
      • Share pulse locations for nearby lights instead of recalculating (grid gen on the fly)

Research

Hue Sync

• Custom palette is nice
• Using audio intensity (onset?) to boost brightness during screen sync

Audio processing research

http://www.cp.jku.at/research/papers/Boeck_Widmer_DAFx_2013.pdf / https://github.com/CPJKU/SuperFlux (normal 2-clause BSD license) / https://github.com/zsiciarz/aquila

Input

Normal input

Will use QtAudioInput for cross-platform capture from input devices (microphones, line-in, ...)

Loopback (capture from system output device)

Windows

Capture from the current output device is possible on Windows: https://msdn.microsoft.com/en-us/library/windows/desktop/dd317585(v=vs.85).aspx

Linux

No idea. QtAudioInput might be able to get a 'monitor' device from PulseAudio, or maybe Linux users can use JACK?

Android

Oddly, Android has an API to capture "low-quality audio", specifically intended for music visualizers https://developer.android.com/reference/android/media/audiofx/Visualizer.html

Processing

Total and frequency-band energy

Onset detection

Dematrixing and 2D locations

[darkzbaron]

Dropped?

[RedSpid3r]

Please can we get this? The screen sync is awesome, so much better than the HueSync App and i'd love to be able to do audio sync, too!

[BradyBrenot]

Project was on hold for a good year, it's alive again, this'll happen eventually.