Support for MediaButton long press / hold / custom click timings - click API is not enough

[sandreas]

Feature proposal

I would like to suggest an extended API for the click(MediaButton b) method, where it would be possible to handle raw MediaButton events (KeyUp, KeyDown) to support MediaButton long press. It could be a method like this:

handleMediaButton(MediaButtonEvent e) {
   // e.Type == keyUp, keyDown
}

Motivating use case(s)

I would like to implement high precision timer events and a MediaButton long press / hold to implement features that are very specific to audio books.

I use an Android Audio Player (HiBy M300 and similar) with dedicated MediaButtons on the device and headset button support.

Unfortunately the current implementation of play/pause, next, previous is not very reliable when using a headset click remote.

What I would like to implement that does not work (like it works in the iPod Nano 7g):

• long press (rewind 30 seconds)
• tap + long press (fast forward)
• double tap + long press (rewind)

The following works (kind of, a bit unreliable):

• tap (play, pause)
• double tap (next)
• triple tap (prev)

To my understanding, the reason, why this does not work, is the handling code here: https://github.com/ryanheise/audio_service/blob/fa845f77654dbbe54d1140c528904509119ed3fe/audio_service/android/src/main/java/com/ryanheise/audioservice/AudioService.java#L985

Only the KeyEvent.ACTION_DOWN is regarded, but there is at least one other event (KeyEvent.ACTION_UP, see https://github.com/advplyr/audiobookshelf-app/blob/fc954a968b47c5f89c171b6cdf22bf38dcc7379a/android/app/src/main/java/com/audiobookshelf/app/player/MediaSessionCallback.kt#L164) that should be regarded.

I think it would be enough to just pass the (mapped) raw event to an audio_service API method to make it work on Android.

What do you think?

[ryanheise]

On Android, you can't intercept a long press anyway since the os intercepts it to open Google assistant.

[sandreas]

On Android, you can't intercept a long press anyway since the os intercepts it to open Google assistant.

Thank you for your quick response.

Well, that may be true for modern stock android devices, but for older or more specialized models this is not always the case. I'm developing for a variety of older devices (Audio players, Phones, Tablets) using Android 7 to 11 as well as custom Android roms (Graphene OS, Lineage OS), where this behaviour is either not available or can be turned off. Even on modern devices this behaviour is not always the default - some of them support long press within Audiobookshelf.

However, even if the long press is not supported, I still think that it would be great to have a more fine grained access to the MediaButton click events - e.g. to support triple or quad clicks or more accurate or customized click timings.

My use case is to recreate the iPod click profile as accurate as possible (because I'm used to it), but I would also like to create custom click profiles in my App, where the user is able to add custom settings for click timings, number of repetitions and as long as it would be supported long press times.

Probably all it would take is a small extension for the listener and one line for android:

    @Override
    public boolean onMediaButtonEvent(Intent mediaButtonEvent) {
        if (listener == null) return false;

        // TODO: use typesafe version once SDK 33 is released.
        @SuppressWarnings("deprecation")
        final KeyEvent event = (KeyEvent)mediaButtonEvent.getExtras().getParcelable(Intent.EXTRA_KEY_EVENT);

        // new code (maybe the platform would also be a nice hint to handle different platforms)
        listener.onMediaButtonEvent(event.getKeyCode(), event.getAction(), "android");
        // new code end

        // ...
    }

See https://github.com/ryanheise/audio_service/blob/fa845f77654dbbe54d1140c528904509119ed3fe/audio_service/android/src/main/java/com/ryanheise/audioservice/AudioService.java#L955

Thank you for making audio_service and just_audio. These are really great.

[ryanheise]

If you can submit a working pull request that most importantly doesn't break the existing functionality, I'll consider it. While click profiles are exactly what I want the flexibility to do, I did look into the long press but ever since Android decided to reserve the long press for themselves, I no longer consider this a personal priority. I'm (personally) fine with following android's latest rules.

[sandreas]

Ok I'll try to submit one, but since I'm pretty new to dart / flutter, it may never happen :)

[sandreas]

@ryanheise So I did some tests trying to implement keyUp and keyDown API methods, but somehow I can't get them to fire. I see the keyUp and keyDown in the debug output, but the events are not emitted within the API.

I hope keyDown and keyUp are not reserved words or already taken in Android.

What I did:

• Cloned the audio_service repository
• Created a feature branch
• Pointed my project to the local repo (instead of the package dependency) via path: ../audio_service/audio_service in pubspec.yaml
• Applied my code changes
  • find every appearance of click and added keyDown + keyUp in the same manner
  • added the call of the API method to AudioService.java for android

There are some missing comment adjustments I'd like to care of, but first I would like to get it working. Here is the current state of the planned pull request: https://github.com/ryanheise/audio_service/compare/minor...sandreas:feature_1068_keyup_keydown?expand=1

Would you mind take a quick look and tell me, what I am missing or if I forgot something?

[ryanheise]

It's not the best answer one might hope for, but I will consider it if/when you manage to get it working (since deprecated features are not something I would personally invest time into.)

[sandreas]

For everyone who is also doing research, here are my findings so far:

• older Android versions show different behaviour on headset clicks
• Android < 7.1 cannot differentiate between keyDown and keyUp events and sends them both together after keyUp with a delay of 0ms regardless how long the headset key is held down
• android >= 7.1 can diff keyUp and keyDown, but keeps emmitting the keyDown event when holding it down every few ms
• on newer Android versions (no exact version, depends on vendor, mostly >=11) the first long press is internally reserved for google voiceover and is hard to override
  • seems that you can override it with extra code and a higher prio VOICEOVER permission in manifest, but i did not manage to get it working yet
• the event emittance is unreliable at best, often double clicks are recognized as one, the more clicks, the less reliable it gets
  • triple click (previous) and double click + hold (next + playPause) are falsely detected in half the cases

However I managed to write a kotlin handler and tweaked the timings as good as possible for audiobookshelf-app. The click detection works ok-ish (accoeding to my logs), but I'm currently still working on the thread communicating with exoplayer.

As soon as I have something working, I'll try to port this to a PR

[sandreas]

My audiobookshelf PR is here - mainly kotlin. I'll wait for the merge and further test results.