Android Auto

[puff]

Adds basic Android Auto and Android Automotive support.

Features:

• Browse & play albums, playlists, and albums in each genre.
• Artists start an instant mix when playing them.
• Jump to albums/artists by letter (click A-Z on scroll bar).
• Seek forwards and backwards in songs.
• Skip to previous and next songs in queue, as well as directly to any song in the queue.
• Shuffle button
• Offline mode support.
  • Even outside of offline mode, it will try to use downloaded items before going online, unless it is fetching all items for root category (e.g. albums, artists, etc).
• Search (voice and keyboard, only online)

Future features/TODO?

• Repeat button
• Favorite button
• Instant mix button
• Home tab
• Genres should start an instant mix (not yet implemented in Finamp)
• Offline mode artists/genres instant mix (not yet implemented in Finamp)

Currently, there is a hard limit of 100 items displayed so the app doesn't crash on large libraries.

// dev notes

• Investigate why it doesn't switch to the Now Playing screen sometimes when starting playback, or takes a while before switching to it. This is likely because of the playback states being wrong.
• Add pagination so we can show more than 100 items. This and This may be useful.

[jmshrv]

This is awesome to see finally :)

Is the 100 item limit just an awkward thing with Android Auto or a thing that specifically needs to be fixed here? Jellyfin luckily makes pagination very easy when fetching from the server.

Artwork sounds like an annoying issue to fix though 🙃

[puff]

Is the 100 item limit just an awkward thing with Android Auto or a thing that specifically needs to be fixed here? Jellyfin luckily makes pagination very easy when fetching from the server.

I put that limit because someone reported Android Auto crashing on large libraries (1000+) I do have an idea on how to add pagination to it, so I'll probably work on that tonight.

Artwork sounds like an annoying issue to fix though 🙃

Yeah, I'm gonna have to do some experimentation with that.

[Chaphasilor]

Thanks for the PR! I can try to take a look at it in the coming days, I just need to find a car to test with ^^

I'm guessing there is no built-in pagination support in Android Auto, and I don't think it would be super useful anyway.
I believe focussing on quick actions, like shuffle all songs, instant mixes and playing recently played items is a good approach.

Search and some voice interactions would definitely be useful down the road (hah), but if basic playback and browsing is working then I guess it qualifies for a beta.
Maybe it attracts some additional contributors :P

[puff]

I just need to find a car to test with ^^

You can use the emulator! https://github.com/jmshrv/finamp/issues/24#issuecomment-1734229067 It's very convenient for debugging too.

I'm guessing there is no built-in pagination support in Android Auto, and I don't think it would be super useful anyway. I believe focussing on quick actions, like shuffle all songs, instant mixes and playing recently played items is a good approach.

Yeah, I think I agree with you. Not sure that pagination should be a priority. I typically just use a playlist anyway lol.

Search and some voice interactions would definitely be useful down the road (hah)

Yeah, 100%. I wanted to incorporate that into this PR, but I was having an issue I didn't really understand where it crashes when searching. EDIT: fixed the crash, was a problem with changing the package name

[Chaphasilor]

Yeah I know about the emulator and have used it last time, but in the end I want to make sure everything works on an actual car too ^^

Regarding voice, how flexible is that? Can it only be used for voice search, or could we enable custom actions, like saying "Start playing Hello by Adele" and it will start an instant mix?

[puff]

Can it only be used for voice search, or could we enable custom actions, like saying "Start playing Hello by Adele" and it will start an instant mix?

I know you can do stuff like that with app actions. There's also a version for Android Auto/Automotive.

[Chaphasilor]

The second link you provided seems to be geared towards navigation/Points-of-Interest apps. Here's the one for media apps: https://developer.android.com/training/cars/media#support_voice

Seems like it has everything we need. If the user indicates an item type (album, playlist, etc.) we can use that to deal with Jellyfin's poor search, and if they omit the type "Play some music" we could resort to shuffle all for now. Would you like to look into it? :)

[puff]

Would you like to look into it? :)

Sure, I have some time the next few days so I'll see if I can understand this and get something usable.

[Chaphasilor]

Awesome. Let me know if you need help with anything!

Edit: Just checked it out on the emulator, here are a few things I noticed:

• Clicking an item in the "Genres" tab should start an instant mix for that genre. Searching for albums by genre is probably not used that often and not really suitable for driving anyway
  • Same thing for artists maybe?
• Placeholder images (especially for artists) would be nice to make sure the item names are aligned
• For albums and playlists I'd really like to either be prompted if I want to play in-order or shuffled (that might not be ideal while driving), or have a way in the settings to set default behavior
• The sorting is a bit different than in the phone app, not sure if you can do anything about it. Names that start with symbols are sorted by the first non-symbol character in the car, but not on the phone. Examples are "...Baby one more time", "& Friends" is "F", ".me", "(Un)Commentary"
• The jump-to-letter function is neat, but only works with the loaded items. Is there a way to load items on-demand, like is done with the new "fast scroller" on the phone app?
• I think we definitely need a "home" tab/section. I'll see if I can get something going for the entire app, since the phone app is supposed to also get a home screen with the redesign
  • There's also a "for you" section in the Android Auto home screen music widget (swiping left on the music widget) where the items from the home screen could be shown. Right now it just shows first 3 items from the albums tab, although strangely in the sorting of the phone app (symbols first)
• There's a display bug when shuffle is active. The order of the list is correct, but the index of the currently playing song is not shown correctly in the list (the bars next to the active song are on the wrong item). Did you do any implementation for the queue? Otherwise I'll have to dive into just_audio once again to see if I can fix it...

[puff]

Thanks for the feedback, I agree with your observations here.

Clicking an item in the "Genres" tab should start an instant mix for that genre

I was thinking we could replace the genre tab with the home tab. Android Auto by default only allows 4 tabs, and genres don't have instant mixes (though we could shuffle albums within the genre).

For albums and playlists I'd really like to either be prompted if I want to play in-order or shuffled

I don't think this is possible, but a setting for the default would work.

The sorting is a bit different than in the phone app The jump-to-letter function is neat, but only works with the loaded items

I don't think I can do anything about these :(, they seem to be built into Android Auto.

There's also a "for you" section...

Seems like this is related to that section, looks to be possible.

There's a display bug when shuffle is active

I've done a little troubleshooting, and I'm confused. It seems there may be a bug somewhere causing a conflict between just_audio and the new queue service? https://github.com/jmshrv/finamp/blob/2ea2156bcbbe78d844997046f7b310539fef53de/lib/services/music_player_background_task.dart#L377

The queue service expects queueIndex to be the non-shuffled, original index: https://github.com/jmshrv/finamp/blob/2ea2156bcbbe78d844997046f7b310539fef53de/lib/services/queue_service.dart#L94-L101

https://github.com/jmshrv/finamp/blob/2ea2156bcbbe78d844997046f7b310539fef53de/lib/services/queue_service.dart#L148-L151 I suspect somewhere in just_audio expects it to be the shuffled index and is sending it to the backend (ExoPlayer?), which results in the queue showing the wrong song as selected. To elaborate, the erroneous selected song in queue is in the position of the original (non-shuffled) index, which is why I suspect this.

Changing queueIndex to the shuffled index, and replacing both occurrences of adjustedQueueIndex in queue_service directly to _queueAudioSourceIndex fixes the issue, however I'm not sure of the implications of this.

music_player_background_task.dart

- queueIndex: event.currentIndex,
+ queueIndex: _player.shuffleModeEnabled && (shuffleIndices?.isNotEmpty ?? false) && event.currentIndex != null ? shuffleIndices!.indexOf(event.currentIndex!) : event.currentIndex,

queue_service.dart

-int adjustedQueueIndex = (playbackOrder == FinampPlaybackOrder.shuffled &&
-         _queueAudioSource.shuffleIndices.isNotEmpty)
-     ? _queueAudioSource.shuffleIndices.indexOf(_queueAudioSourceIndex)
-     : _queueAudioSourceIndex;
+ int adjustedQueueIndex = _queueAudioSourceIndex;

[Chaphasilor]

We could reduce the number of tabs even further, with a home tab and a library tab? On the library tab we could have a grid with the different categories.

Genre mixes should be possible, at least there's a way to add a genre to a mix on the phone 🤔 But maybe that just adds all individual albums, not sure.

Can we change the sorting within Android Auto? If something like "Recently Added" is possible, then it should be possible to use the BaseItemDto's SortTitle too.

I'll take a look at the modifications to the queue service. just_audio is really opinionated and the layered architecture doesn't make things easier. I've struggled with it for many monhs 🙃

Edit: Genre instant mixes should be a thing, at least it works in the Jellyfin app. No idea why it's hidden in Finamp...

[Chaphasilor]

@puff I managed to fix up the queue based on your proposed changes. There were a few other things I needed to adjust, and I fixed a bug that I introduced a while ago (items in next up were not being reported to the external/OS queue).
From my testing everything seems to work as it should, but please let me know if anything is off!
With the beta release approaching fast I would hate to break things, and there have been the strangest bugs with playback and queue management in the past...

[puff]

We could reduce the number of tabs even further, with a home tab and a library tab?

Yeah, I like that idea.

Genre mixes should be possible

I didn't see a way already implemented in Finamp yet, but yeah it should be possible if the Jellyfin app has it.

Can we change the sorting within Android Auto?

Yes, we can sort the MediaItem list before returning from getChildren. While testing, I noticed the offline sort already used doesn't match output from the Jellyfin api when sorting names. https://github.com/jmshrv/finamp/blob/4463bff0998f22c4558f1178d9d90c399527f090/lib/components/MusicScreen/music_screen_tab_view.dart#L327

Looks to be because the Jellyfin api is case-insensitive: https://github.com/jellyfin/jellyfin/blob/7027bc00fc06314929011735e425763779cb4076/MediaBrowser.Controller/Entities/BaseItem.cs#L897

I've fixed it and added sort to Android Auto. It uses whatever sort option is selected in the phone app. Before & After:

[Chaphasilor]

Thanks for fixing that. There are some sorting-related issues that need to be tackled at some point anyway, like #444 or #581, so if you feel like taking a look while you're dealing with sorting anyway, that would be much appreciated.
Otherwise I'll try to fix it after some bigger issues have been fixed...

[puff]

I managed to get artwork to work with offline items and (kind've) Android Automotive using the android_content_provider package.

The white icon in the middle is the placeholder art.

Only problem is that for Android Automotive, we can't use http(s) URIs. Because of this, we have to resolve them ourselves. My current implementation hangs until every item has a resolved image. I think the way to fix this is to offload it to the ContentProvider. For now, I've commented it out until I have a better solution.

[Chaphasilor]

Did you manage to take a look at the voice command stuff? Or is there anything that you need help with? :)

[puff]

Did you manage to take a look at the voice command stuff? Or is there anything that you need help with? :)

I've been having some trouble with it. It doesn't default to finamp even when it's open (maybe I forgot to configure something though?). It really doesn't like to recognize the app's name, you have to say it slowly. Saying "album" or really anything else other than the name of the song/album makes it default to YouTube Music (probably your default music service) so extras is always null.

"play some music on finamp" Seems like some generic phrases are stripped out, odd that the "on finamp" part wasn't though.

"play black ray on finamp"

If you want to try it out, you just need to add this to the manifest:

<intent-filter>
  <action android:name="android.media.action.MEDIA_PLAY_FROM_SEARCH" />
  <category android:name="android.intent.category.DEFAULT" />
</intent-filter>

and uncomment playFromSearch in music_player_background_task.dart

[Chaphasilor]

Sorry for not responding earlier, I was a bit busy. Just tested it out and didn't manage to invoke the playFromSearch() method even once, although I did manage to get it to recognize the app's name a few times. I did manage to open the app by saying "Play some music on Finamp", but I don't think that's related to the manifest changes.
If the problem would just be to properly recognize the app name, then this might help, if it still works. But I think the whole voice command stuff seems to be pretty flawed, with YouTube being the default for a lot of things even after explicitly not selecting a default in the Assistant settings.
I only tested with Google Assistant, not with Android Auto, but I'd expect results to stay the same.

I guess we should skip this feature for now, including regular search. Search is due for a rewrite soon, and it makes little sense to integrate the old version now...

[puff]

Thanks for trying it. As for the link you provided, it looks like the app may need to be published on Play Store in order to access app actions and change the pronunciation.

I changed the default music service in the Assistant app's settings to "No default provider" and it now defaults to finamp instead of YouTube Music when you say "play album x" or "play some music". This isn't really ideal though because users would probably have to do this too. The extras is null because audio_service doesn't actually pass it to our callback. Someone made a PR for it here: https://github.com/ryanheise/audio_service/pull/1015 It also doesn't pass it to some other callbacks, but that's another issue ig :P Might be nice to have extras because Google sometimes provides us the album and artist name. (Note for future: don't rely on this 100%, it should search for query if it cant find it with given artist/album name)

Also, I found out that Android Auto apps need to support voice commands to be published on Play Store.

[Chaphasilor]

They have to support voice commands? o.O
Could you share a link to the relevant docs? Is there a specific type they need to support, and does it have to go through the Google Assistant, or would speech-to-text for search be enough?

Requiring voice commands, which currently aren't really working, kinda sucks. I guess we could just throw it in the manifest and pretend like we support them, but I'm not sure.
What are your thoughts on this? Would you prefer merging some form of this quickly, or do you want to spend more time on this and try to figure it out fully? :)

[puff]

Could you share a link to the relevant docs?

https://developer.android.com/training/cars/media#support_voice https://developer.android.com/docs/quality-guidelines/car-app-quality#app_categories

This and some github issues suggest they require voice commands. https://github.com/signalapp/Signal-Android/issues/6124 https://github.com/ryanheise/audio_service/issues/956#issuecomment-1635113977

I guess we could just throw it in the manifest and pretend like we support them, but I'm not sure

I don't think so, from the above github issues it seems they test them.

Maybe we could disable Android Auto (comment it in manifest) in the Play Store release until we have it implemented? I would like to get it working though so we don't have to... I'll try implementing the search algorithm for it.

[Chaphasilor]

Okay, got it. I still can't get a simple "Play [song title]" to work on Finamp. It still defaults to YouTube on my phone (although it fails to actually play), and appending "on Finamp" at the end makes it open YouTube and actually start playing, super strange. Did you try it through Android Auto or also just with the regular Assistant?
As long as we can get a simple "Play [song title]" command to work, we should be fine.

Here's my quick implementation of how the search could work, it starts an instant mix for the first match returned by Jellyfin's search:

  @override
  Future<void> playFromSearch(String query, [Map<String, dynamic>? extras]) async {
    _audioServiceBackgroundTaskLogger.info("playFromSearch: $query ; extras: $extras");

    final jellyfinApiHelper = GetIt.instance<JellyfinApiHelper>();
    final finampUserHelper = GetIt.instance<FinampUserHelper>();
    final audioServiceHelper = GetIt.instance<AudioServiceHelper>();

    try {
      
      final searchResult = await jellyfinApiHelper.getItems(
        parentItem: finampUserHelper.currentUser?.currentView,
        includeItemTypes: "Audio",
        searchTerm: query.trim(),
        isGenres: false,
        startIndex: 0,
        limit: 1,
      );

      if (searchResult!.isEmpty) {
        return;
      }

      _audioServiceBackgroundTaskLogger.info("Playing from search query: ${searchResult[0].name}");
      await audioServiceHelper.startInstantMixForItem(searchResult[0]);

    } catch (err) {
      _audioServiceBackgroundTaskLogger.severe("Error while playing from search query:", err);
    }

    return;
  }

It's untested, since I could never get it to be called...

[puff]

Did you try it through Android Auto or also just with the regular Assistant?'

Android Auto. I tried with the Assistant app on phone and I get the same issue as you. Using the media controller test app (MCT) allows you to call playFromSearch directly though.

When using voice commands, try also saying "song" before the name like "play song x."

Your search code does work for songs. However, the test in the MCT app sometimes fails. Probably something with async stuff. I'm not sure if this matters though, because the song does actually play.

[Chaphasilor]

Would you mind sending me an APK for the test app? I couldn't find a ready-made one and am not super familiar with Android Studio :)

I'll try your tips and take a look at the search issues. I think it's expected to use different states while preparing playback, and that is probably missing in our case. Not sure if there's anything we can do about it though...

[puff]

Sure, here's a compiled apk for the MCT app mediacontroller-release.zip I used IntelliJ IDEA to compile it.

[Chaphasilor]

After playing around with this, I can confirm that playing from search works. In Android Auto it works by default ("play song xyz"). I also observed that Google Assistant seems to "enhance" search queries in some cases: "play 404" results in a response "okay, asking FinAmp to play 404 by Knife Party". I'm pretty sure the artist name is not coming from Finamp in this case, but I could be wrong.

I found no way to make the tests pass by fixing the state. There are some functions within audio_service that should be working, but don't really do anything, for example setting the queue title. Since we have a queue source, it would be nice to show it, but adding a value to queueTitle within the BaseAudioHandler doesn't work.

I also tried implementing the regular search after reading that we can show search results that the user can choose from in case the wrong item was picked, which isn't uncommon with Jellyfin's crappy search. I managed to get the search results to show, but selecting a result doesn't actually start playback yet. I'm working on fixing that though :)

[Chaphasilor]

Okay, managed to get search and voice search alternative results to work:

https://github.com/jmshrv/finamp/assets/18015147/858086e1-bc2a-447e-85db-7f365a8d8b56

I guess the only thing that we need to do now is try to get any many of the Android Auto tests to pass as possible. The main issues here seem to be "skipToNext"/"skipToPrevious"/"skipToQueueItem", and the media artwork test. Not sure if browse tree depth and custom action icon type are actually required.
I might replace the custom action icons at some point, so that should fix one of the tests.

One thing I'd like you to take a look at is improving the handling of the mediaIds. I know it sucks that we are only passed the mediaId and nothing else, but maybe it's possible to get the required into about the "category" from somewhere else instead, like the context / active tab (or lack thereof)? If not, I'd still like to replace the "-1" (and now "-2") special IDs with some kind of enum that can be stringified and reliably compared, and comes with additional meaning. Maybe instead of splitting the ID you could also simply use a JSON string with custom fields instead, to "properly" pass all the required data? Would be great if you could take a look at it :)

It really seems like we're slowly but surely getting there! :D

[Chaphasilor]

@puff so the PR you mentioned go merged yesterday and I just played around with it. We get the extras now, and I implemented playing back songs, albums and artists now. I think that's all the types that we can differentiate, playlists don't seem to work.
Albums are played in-order (even if I say "shuffle album xyz..." we don't get any extras provided to indicate that) and artists play as instant mixes.

Song and album results are also (rather crudely) filtered by trying to match the artist provided by Google Assistant to the search results. This does work surprisingly well, and is kind of necessary because whenever GA knows a song, it replaces the query with "<song name> <artist name>", breaking Jellyfin's search and making the "Search results" button mostly useless.
The problem here is that while playFromSearch() receives the proper extras, search() does not, so we cannot tell what the title and artist are. So I tried to get the best match possible right from the initial voice search by using the provided metadata.
Take a look at the commit message for the necessary syntax for requesting different items.

Edit: before I forget it (for the update notes): songs can also be played by saying "play <song name>", and if Google Assistant fucks up and converts that into some other song that wasn't meant, saying "play song <song name>" does help in most cases.
Edit2: If not, "play the song called <song name>" might help.
Edit3: Same applies to playlist, "play <playlist name>" might work, otherwise try "play playlist <playlist name>", otherwise "play the playlist that's called <playlist name>".

[Chaphasilor]

@puff are you still around? I can handle the refactoring, but is there anything else that needs to be done before this can be merged and released? 😅

[puff]

Sorry for the (very) late reply!

maybe it's possible to get the required into about the "category" from somewhere else instead

I don't think it is. An enum sounds like a good idea though.

try to get any many of the Android Auto tests to pass as possible

I think these only fail because of the playback state not being set in time because the functionality works.

There's also a "for you" section...

Before, I said this might be possible, but it's not because we need onGetRoot which isn't exposed by audio_service.

is there anything else that needs to be done

~~The only thing I know of is the bug with artwork on Android Automotive (not Auto), it doesn't display artwork for online items because we have to resolve the image ourselves.~~ FIXED

Pagination can be added later, I don't think it's necessary to be able to scroll more than 100 items. Maybe a configurable limit in settings though? Note: This and this seem useful for pagination.

Also, fantastic work on the search features!