Draft for playon compatibility with the jellyfin server

[pinsarda]

This is a very early shot at getting the playon feature from jellyfin to work with finamp. For now this is a extremely rough draft to start working on the feature and getting rid of an annoying bug. Can only start songs without having media controls and need to enter the url manually.

[Chaphasilor]

Thanks for getting this started, really looking forward to it!

So I managed to get this working a bit more reliably and generically some time ago, just pushed those changes :)
The playback controls are now all shown, and Finamp registers support for all commands I could find. The important ones are already implemented too.
I didn't work on the actual "play on" functionality though, so that's unchanged, meaning it works for the most part, but sometimes the wrong item is picked (e.g. selecting a specific track from an album will play the first track from the album instead). Ideally, this should work just like it works in Finamp, and make use of the same settings. So, if a single track is selected, depending on the Finamp settings it should either play just that track or start an instant mix from that track.

There are still issues with the websocket, it seems to get disconnected (or at least stops receiving messages) after a short while or certain commands. Haven't looked into that just yet.
I also noticed when testing this out again that when logging into the server the websocket connection will fail until the app is restarted, because the connection is established before the URL and credentials are know. The initialization should therefore be delayed until setup is complete, or restarted after successfully logging into a server.

[pinsarda]

I apologise for taking so much time to go back to this, but as they say better late than never ! I didn't find the source of the issues with the websocket, still investigating. For now I quickly fixed the item selection, now it starts the instant mix and the correct song as it should.

[pinsarda]

The current implementation works very well on my phone, but can probably be a bit slow as I request each song individually instead of the whole album at once. I don't know if there's an easy way to do it better currently...

[Chaphasilor]

Well just take a look at how the content of an album is fetched for the album screen :)
It's just a matter of setting the parentID filter, I think.

[pinsarda]

Yes I thought about that but this would introduce a bug when starting a playlist. The websocket just gives us ids of songs to play. It doesn't ask finamp to play an album or a playlist. This makes it hard to do it differently.

[Chaphasilor]

Is there no way at all to differentiate from "where" the track was selected? I guess in that case all we can do is play just that one track (or start the instant mix, depending on settings), and nothing else.
Is that also the case when you try to play an entire album or playlist? Does it only send us the first track, without any additional information?

[pinsarda]

I don't think so, we just receive a list of ids, and optionally the index of the track selected if there was any. The current implementation does what you're proposing. If a specific track is selected, we only play this one or start instant mix. And if there's no precision on the starting index it queues the whole album/playlist. After testing it seems like the best possible behaviour to me.

[pinsarda]

As discussed in https://github.com/jmshrv/finamp/issues/500 an internal volume controller will be implemented eventually, we'll rely on that instead of system wide volume.

[Chaphasilor]

@Maxr1998 how did you approach this issue (playing albums/playlists) on the Android (TV) app? 🤔

[Maxr1998]

Can't speak for Android TV, not even sure play on is supported there. On the Android app, we just wrap the web app so the handling is identical to that.

[Komodo5197]

This seems like the server is giving us a list of song IDs to form the queue, which means the most similar piece of code is loadSavedQueue() in lib/services/queue_service.dart, so you can probably look at that for inspiration. Most notably, you can make batch requests for up to 200 item IDs at once, which is significantly faster. Also, I'm pretty sure the behavior of loading the single song or an instant mix is questionable. We should be loading the whole id list from the server into the queue and just setting the initialIndex to what the server tells us to.

[Chaphasilor]

@Komodo5197 the playing of a single track should only happen when the user selects a single track in the other client, and the API only gives us one ID. If we get multiple IDs, we should play all of them of course.

Thanks for the suggestion about the queue restore, that would make sense.

However, my main concern was how to identify where these tracks are coming from, so we can show a proper queue source. If the user requests to play an album, I'd like Finamp to show which album is playing as the queue source name (so that one could also tap the source at the top of the player screen, and immediately go there). Same for playlists, artists, etc. But it seems that info is simply not given to us, so we'll have to default to some generic ("Queued by Remote Device"?) name instead.

[Komodo5197]

We are getting the full queue. I click a track partway through an album on the web UI and finamp recieves {"MessageType":"Play","MessageId":"560c155a593a497e8bd047610f2e8e4a","Data":{"ItemIds":["46430779c8072941ed8f7e2ddafaebe4","f8ee032bb75db53c1c317c572e3e8814","fda9dba1e21b261bdb050efe5bd95bbb","4b0a4b5c756079eb8322860215fbe5fb","e012de5b9fb821a2258b6e91c9225459","ea8e6422f6266c453469e104e3e4b950","cbb922f4f858571b79dafa8da3578270","d891ee5acb57e58ca338df5518c8ce37","80ed7b24b4878c28a75e6234ff066bd6","ca4e6f99c7ad2bae487aaba0585f869c","d586e2361bab084b04872a4128507db8","67f19176d6582739abfd0678442b45de","c34b8e312fd99b7194d927c64a9700a9","c89368641479249da0b333c09ba1aa40","ba8ade9d002d8c40bae852531bcebba5","8ac7aa4480fa7fcaecf1d03a816431ed","22a0587b3018e44644924535542ace41","ca63bf02ba1a0bee484e2fb6e036d3e0"],"PlayCommand":"PlayNow","ControllingUserId":"de6a683bafb44247881bb62bf125f7e8","StartIndex":8}}.

There are also additional play commands 'PlayNext', corresponding to addToNextUp(), and 'PlayLast', corresponding to addToQueue(). Implementing those allows remote queue building that seems to match up with how the web client works.

Regarding queue source, it does indeed seem that we don't have this information in any form, and will need to use a generic header.

Also, I happen to still be running a 10.8 server, and to get this to work I had modify the ClientCapabilities to explicitly set supportsSync and supportsContentUploading to false, and remove the supported command "SetPlaybackOrder".

[Chaphasilor]

Okay, thanks for taking a look. Haven't had a look at the code recently, but there are still instances where a single track is selected, e.g. when playing from a search result. In that case, the Instant Mix behavior should still apply, I think.

Interesting note about the 10.8 server. I also still have one of those lying around, I'll give it a try myself. Maybe the commands had a different name back then? I'd like to keep the playback order functionality if possible, but I'd also like to keep supporting 10.8.
Maybe it's finally time to query the server info (version, name, etc.) on startup, and have some version-specific features. I guess feature detection is a hopeless endeavour with Jellyfin anyway...

[Chaphasilor]

I gave the current version a try yesterday. It was working well at first, and I could see the queue, playback info, and control playback mostly fine. But later on I didn't manage to get the queue or even the playback info to show up anymore, even after restarting the app.
I also encountered an issue where it seems like commands where being queued for some reason. I tried playing/pausing and sending a few other commands (including a message) through the web client to Finamp, which didn't have an effect. But then many minutes later I tried to pause playback (in Finamp directly), but it was resumed immediately. That happened a few more times, and then finally it actually did pause and stayed paused, and then the message appeared. Maybe there's a command waiting for a Future that never resumes, or something similar?

[pinsarda]

For the first issue, it's weird I didn't experience it. I got the same bug regarding command queuing though, and got to the same conclusions as you without nailing it yet. I'll get back to it tomorrow and on thursday. I'll also adjust the code to have the behaviour proposed by @Komodo5197

[pinsarda]

@Chaphasilor Couldn't spend as much time as I wanted on this, but as we guessed, using unwaited instead of await when executing the commands (expecially Pause, Unpause and PlayPause) completely fixes the issues I had with unresponsiveness after sending too many commands, and also makes everything smoother. Is it a bad practice to do so ? It definitely needs some more testing to be sure but now it seems to "just work" on my device. There's one remaining bug when seeking I have to fix. Jellyfin takes a bit of time to update the progress bar on the web client, because finamp doesn't report playback state directly.

[Chaphasilor]

Hmm. It's good that we seem to have found the issue, but the best idea is probably to figure out why the play/pause futures don't resolve like they should.

Aside from that, will the websocket reconnect to the server after changing i.e. from WiFi to mobile data?

[pinsarda]

Ok, I'll check further, I didn't see anything suspicious last time I checked the futures. With my current setup (VPNs and stuff) I unfortunately can't check easily if the websockets reconnect when changing network, but I highly doubt it. Disconnecting and reconnecting wi-fi closes the connection. Changing this, as well as fixing the bug when login in for the first time, is on my to do list

[Chaphasilor]

Just wanted to note that the websocket connection should get disconnected when switching to offline mode, before I forget about it.

[pinsarda]

There's an issue when using playlist, the songs are completely unordered. This is an upstream bug from jellyfin : https://github.com/jellyfin/jellyfin/issues/8885 I didn't look into the details just yet

[Chaphasilor]

That's because the server sends us the raw track list instead of just the playlist ID, right? If there is an ID / some way to figure out which playlist is being requested, we could simply disregard the track list and fetch the playlist ourself?

[pinsarda]

Absolutely, but the playlist id is not in the message received by the websocket unfortunately. Plus you have no way to even know if it's a regular album or a playlist being played (apart from checking if all the songs belong to the same album after fetching data). As you guessed, we only get a list of ids of songs to play and a starting index, that's it. I have dirty workarounds in mind but I would rather have the bug fixed properly jellyfin side. And/or have the playlist/album id supplied directly in the websocket data, which would also be nice to populate the "playing from" label.

[pinsarda]

In the last commit, I am not too sure about my "updateState" function in favoriteProvider, I couldn't find an other way to do it directly using riverpod's ref. It works but may be unnecessary to create this function.

[Chaphasilor]

Changes look good! I'll probably go over the exact wording an capitalization tonight.
Now, what is left to do before this can be merged?

I can think of:

• [x] Make sure output panel doesn't throw errors on non-Android platforms'
• [x] Report queue to server by default
  • Maybe we even want to add a setting for disabling the Play On websocket connection? Are there any downsides to it (probably battery and data usage)? Then we could automatically send the queue if Play On is enabled, and otherwise use the state of the dedicated setting for sending the queue
• [x] (Potentially) Integrate changes from #1100 to use the new internal volume
• [x] Test on desktop

Improvements for Future PRs:

• [ ] Link volume slider in output panel with system volume by default (on mobile), but allow detaching it for controlling the internal volume.
• [ ] And of course, implement controlling other clients

Also, for the release notes I'd like a list of notes/requirements for using this:

• Jellyfin bugs to consider (only admin can control, playlists in wrong order)
• Ideal settings to make sure it works reliably (e.g. disable all battery saving
• Supported Features, but more importantly unsupported features
  • Switching tracks within the queue is still unreliable
  • Loop/shufflle can't be toggled yet

Please add any points you can think of :)

[pinsarda]

On linux desktop it works like a charm. I'll add the toggle to disable the feature completely in the settings today when I have 2 minutes. For the ideal settings to use, I think it needs testing. I should beta test a release build.

I don't have much to add, you already listed a lot of things, but :

• We discussed android auto disabling the feature, do we still want that ? If so I looked into it briefly, but can't get a boolean or settings listener to catch this event.
• A common issue that could arise is using a misconfigured reverse proxy that blocks websockets (nginx does it by default afaik)
• For jellyfin web bugs, sometimes (rarely) the player seems to work unreliably, but a simple socket reconnect from the remote client does the trick in my experience

[Chaphasilor]

Okay, the reverse proxy stuff is a good point!

[Chaphasilor]

ah one more thing, I noticed that the landscape mode breakpoint is a bit broken, so I still wanted to fix that. probably due to the new output button.

[Chaphasilor]

@Komodo5197 since you recently removed the postLaunchHook, where should I best initialize the Play On service now? The main challenge is obtaining a WidgetRef so that we can update the favorite state...

[Komodo5197]

We could grab the providerScope from one of the global snackbar keys and use that. That would allow us to set this up from main with anothrer user login hook. Alternatively, it looks like we could manually create a global ProviderContainer that we register with GetIt. Then we can replace the top level ProviderScope with an UncontrolledProviderscope using that container, and any service code could use the global container to read or listen to providers. Also, you may want to have a check in the favorite provider update where if the old and new values differ, call ref.keepAlive(); and _changed = true;.