Add support for haptic playback in the ExoPlayer

[robot-controller]

Use case description

Haptic feedback is an important feature of many modern Android application, as it can greatly enhance the user experience. Furthermore, haptic feedback is often accompanied with audible feedback, sometimes even a visual animation.

For optimal user experience, it is vital that all channels of these multimodal (tactile + audible + visual) experiences are synchronized. Specifically the tactile and the audible signals require a precise synchronization, while human perception allows for somewhat more tolerance regarding the visual signal.

A way to play back rich haptic feedback in a way that's synchronized to audible feedback would solve this need and enable greatly enhanced tangible interactions.

See https://source.android.com/docs/core/interaction/haptics/haptics-ux-foundation#tips-for-implementing for more details.

Proposed solution

The MediaPlayer and the SoundPool APIs support playback of .ogg files encoding both audio and haptic channels. It would be very useful to have support for these .ogg files in the ExoPlayer as well.

Alternatives considered

The MediaPlayer and the SoundPool APIs support playback of Audio-Coupled haptics. However, since Android 16, unreliable behavior of the MediaPlayer has been observed (see https://issuetracker.google.com/issues/433531614). Furthermore, the usage of the ExoPlayer is preferred over the MediaPlayer in general, according to the documentation here: https://developer.android.com/media/platform/mediaplayer.

[oceanjules]

Continuation of https://issuetracker.google.com/issues/433531614. Assigning to @icbaker.

[icbaker]

Internal reference: b/434863998

[krocard]

The old, Ogg haptic track

As far as I know, the OGG "haptic" audio track was never meant as a public feature, it was designed for device specific ringtone files. A symptom of this limitation is that the haptic track needs to to be tuned by device, only ogg is supported (the ringtone format), only mediaPlayer is supported...

The new, HapticGenerator

The recommended way to have audio synchronized haptic is to use the HapticGenerator (as you already found). Nevertheless, it has a fundamental limitation: it doesn’t allow any control. The played audio is converted (through various audio filters) to haptic without any app customization.

Not enough

Media aware haptic requires to only hapticaly enhance specific events. For example vibrating on shots in a video game, base drum in music, impacts in sports... See for example the new Formula one iOS trailer. Indiscriminately vibrating the device on low frequency sounds like the default HapticGenerator does is not enough to offer an immersive haptic experience.

Workarounds

This doesn’t mean that nothing can be done using the existing SKD APIs:

1. Calling Vibrator.vibrate() at the right time, is the easiest workaround, but it isn’t real time enough. Nor does it offer much haptic pattern.
2. A better workaround would be to play a second AudioTrack with just the audio that should be processed by the Haptic Generator:

Player --- "Audio"  AudioTrack --------------------- mixer ----  speaker
      `--- "Haptic" AudioTrack -- HapticGenerator --´
                                           `--------------------- haptic motor

Nevertheless this is far from easy, here are some expected challenges:

• how to start in sync the AudioTrack? (is it enough to start them at the same time, do they need speed adjustment)
• can the haptic audioTrack be muted so that it doesn’t play to the speaker?
• Or should the haptic "audio" be subtracted from the main audio (as they would be added in the mixer). This would require sample accurate sync to avoid glitches.

The future?

I believe that any significant improvement to the status quo would require to modify the Android audio framework:

One way to implement haptic playback would be to add a audio haptic channel type to AudioTrack (eg: left+right+haptic). In practice that means adding an AudioTrack haptic channel mapping. It would only be used by the HapticGenerator and dropped by the audio mixer. While this would require changing AudioFlinger, though it should be fairly straightforward.

[icbaker]

Thanks for the write-up Kevin.

One way to implement haptic playback would be to add a audio haptic channel type to AudioTrack (eg: left+right+haptic).

I think this channel config already exists, and is used by the MediaPlayer ogg haptic handling - it's just not public (or in Java) so we can't directly use it from ExoPlayer.

My (untested) assumption is that ExoPlayer could match MediaPlayer support for ogg file haptics by:

1. Parsing the metadata from the ogg file that signals the presence of haptic info
2. Setting the appropriate channel mask on AudioTrack and unmuting the haptic channel using AudioAttributes

Although of course this doesn't solve this part:

A symptom of this limitation is that the haptic track needs to to be tuned by device

But that sounds more like a content-generation issue, and/or a limitation of ogg format specifically - which I imagine could be resolved by a richer format in future with more tuning metadata - and ExoPlayer could update it's extraction logic to support it.

Given the tight synchronization needs between audio & haptics, it seems reasonable to me to continue with the approach of a single AudioTrack that handles both (rather than trying to synchronize two AudioTrack instances).