The Playback Rewrite

[nielsvanvelzen]

:rocket: What is The Playback Rewrite?

The goal of the rewrite is to create new playback code from scratch for video playback, audio playback and live tv. The new code should be easier to maintain and be prepared for a migration to Jellyfin 11. New features like SyncPlay can be added without much trouble. All code is Kotlin-first and uses the new SDK. It should be used by all our official Android apps for consistent behavior between (Android-) devices when playing files.

The rewrite is a very long project because I'm just one person volunteering to work on Jellyfin in my spare time. There is no hard deadline or specific version goals. This issue contains a task list of the bigger features/changes.

Status updates will be posted occasionally in the comments. Please read the bottom paragraph of this issue before commenting yourself.

:trophy: Tasks

This list will change over time, some features might not be available on the initial release of the new playback code. When possible issues/pull requests will be linked for tracking purposes. There is no particular order for this list, new features might be added before restoring older features.

The amount of checked boxes does not indicate progress or completeness of the rewrite. Some changes required to make this happen have been running in production since version 0.15.0.

:building_construction: Preparations

Various tasks that need to be completed to work on the rewrite.

• [x] Create new Kotlin-first SDK
• [x] Add new SDK
• [x] Support multiple Gradle modules when building - #1072
• [x] Add a way to enable the new playback module - #1064

:fireworks: Proof of Concept

• [x] Design architecture - #2401
• [x] Add playlist support - everything is a playlist, even single item playback - #2401
• [ ] Add bare-bones player UI
• [x] Add ExoPlayer backend - #2427
• [x] Load video from server
• [x] Report playback progress to server - #2428

:arrows_counterclockwise: Restoring features

• [ ] Support transcoding/remuxing
• [ ] Add LibVLC backend
• [x] Add remote control API - #2428
  • [ ] Support multiple items in PlayMessage #2394
• [ ] Add intros
• [ ] Add chapter support
• [ ] Move "next up" screen to new playback code

:hammer_and_wrench: Integration (ATV)

• [x] Use new playback module for audio (music, podcasts, audiobooks etc.) - #2453, #2455
• [ ] Use new playback module for video (movies, series etc.)
• [ ] Use new playback module for Live TV
• [ ] Delete old playback code

:hammer_and_wrench: Integration (mobile)

• [ ] Make player UI touch compatible
• [ ] Release playback module to Maven Central (same versioning as the app)
• [ ] Replace native player with playback module
• [ ] Restore missing features

:boom: Adding new features

• [x] Add smart codec detection for directplay, remux and transcoding
• [ ] Add system preferences to overwrite codec support manually
• [x] Add playback speed controls
• [x] Add Media Session integration - #2457
  • [ ] Use ImageProvider to proxy images
• [ ] Add Picture in Picture support
• [ ] Add gappless playback - #2427
• [ ] Add SyncPlay
• [ ] Add player stats (similar to YouTube's "stats for nerds") for debugging

Server blockers

Some features cannot be made or won't work great without changes to the server. This is a list of them.

• [ ] jellyfin/jellyfin#8971 Required to properly implement speed controls, it would also need changes to the dashboard of the web client
• [x] jellyfin/jellyfin#6052 Required to implement SyncPlay

:exclamation: This issue

Please do not reply to this issue for any questions about the playback rewrite. The comment sections is meant for our contributors to give updates or discuss the implementation. For anything else, please create a new issue or discussion. We may delete your comment if we consider it off topic.

[mueslimak3r]

I was looking intro the code for trailer playback last night and I see that the object for external trailers just uses links videos (eg: a link to a youtube video). I'm not sure how feasible it would be, but it seems like a good design for a re-written player might be for it to just play video links. Then other classes would be queried for stuff specific to jellyfin like subtitle object classes, mediastream info, etc.

With the current player it's entirely designed around using jellyfin specific classes like BaseItemDto and its MediaStreams and SubtitleInfos, etc. Because of this it's not possible to make exoplayer play an arbitrary youtube link, even though exoplayer is capable of doing so.

[nielsvanvelzen]

it seems like a good design for a re-written player might be for it to just play video links.

The current implementation already does this, it never uses the Jellyfin API directly to make it easier to update if the ever API changes. This doesn't mean that YouTube video is supported though, the implementation will only allow video from the Jellyfin server. If we want to support YT a server plugin should deal with that, not the client.

Then other classes would be queried for stuff specific to jellyfin like subtitle object classes, mediastream info, etc.

My idea is to create some kind of "lookup" service where the playback code queries for data. This service would be implemented in the app code. I haven't worked on this part yet.

it's not possible to make exoplayer play an arbitrary youtube link, even though exoplayer is capable of doing so.

I don't think this is possible without using the proprietary YouTube API.

[Andy2244]

Just a quick note regarding your thoughts on "Rethink external player support".

The most interesting players that have better or more features than Exoplayer nearly all have a basic API, so its mainly a matter of implementing the available api. So my suggestion would be to define a clear interface that can be implemented for a particular player. So as example there is often no need to use a external player for audio, while you do want to use it for video. So having a interface to probe what is implemented and whats not may help.

So from my perspective Exoplayer/libVlc should be handled like a external player, but with fully implemented interfaces as reference.

Just a suggestion.

[nielsvanvelzen]

Quick update for everyone watching the progress on this rewrite. I'm still working on it! Even though the issue was made quite a while ago I've made a lot of progress but also started over a few times. I want to get the basics right so we don't end up with unmaintainable code.

My focus is currently on getting audio-only media to work perfect before proceeding with video support. This will probably mean that eventually the app releases an update with the new playback code for music/audiobooks and the old playback code for series/movies.

The following list of features are working locally already, there might even be more I forgot to mention:

• Queueing with lazy loading
• Media session support (the notification on the home screen for what media is playing, this also allows controlling via the google assistant / smartwatches etc.)
• Jellyfin remote control
• Jellyfin Playstate (showing what is playing in the dashboard, marking items as watched/listened etc.)
• Basic audio playback (no transcoding)
• Gappless playback

There's still a lot of bugs and some design decisions to make in the overall architecture. Hopefully this it won't be too long before the first pull requests with actual code go up. Thanks everyone for waiting!

[gitdeath]

Would the Media Session Support mentioned be the solution for #441? If so, is this something that could be included in one of the 15 releases, or is it gated behind other things in this project?

[nielsvanvelzen]

Yes media session support will fix #441 and no they're not easy to integrate in the existing playback code. If everything goes right the rewritten playback code will be available in 0.16 for music playback (no promises).

[nielsvanvelzen]

I've updated the initial post; removed some unnecessary info. Rewritten the intro and added a list of changes required in the server.

The past few weeks saw quite some progress on the rewrite (see the linked PR's above this comment). The current plan is to use the new playback code for audio playback (music/audiobooks) first before working on video again. This is mainly because it's easier to get the architecture started and have some working code.

Music playback is already working for me locally and I'm slowly cleaning all code and making pull requests to get it into the main repository.

Thanks for being patient with my 2 year rewrite project 😄

[nathangur]

Yeesh. 2 years ago and 12 have been finished

[nielsvanvelzen]

Hello everyone, I'm back with another status update!

The last few weeks saw a bunch of changes (see pull requests above this comment) and I can now say: we have reached feature parity with the old music player 🎉. Today I merged a change that uses the new playback code as default for music (and other stuff using the audio player).

Work will continue to add additional features, improve performance and fixing bugs. I will also accept pull requests to the new playback code now. Please discuss first with me if you're planning to make changes.

When 0.16 is released I will start work on video playback so we can hopefully fully replace the existing player code in 0.17/0.18 (as always; no promises). I plan to start the beta cycle for 0.16 in the coming months.

PS: a quick reminder to read the issue description before replying, your comment is likely to be removed/hidden. Also read our Code of Conduct before complaining to unpaid volunteers.

[nielsvanvelzen]

The first beta for 0.16 is here!.

As said in previous posts, this release uses the rewritten playback code for all music playback by default. Like all beta releases it will be available in the beta program on Google Play only when it's been trough review.

If you encounter any issues you can (temporarily) switch back to the old backend from the app preferences, please report your findings on GitHub (in a new issue) so I can fix the bugs before release.

I'm already working on the initial changes for video support locally which I'm hoping to make a part of 0.17 (no promises!) which I'm hoping to release before the end of this year (also no promises!).

If you appreciate my work, you can support me by donating via GitHub sponsors. I use these sponsorships to buy equipment for my development machine and hardware to test specific issues.

[thor2002ro]

even if av1 still doesnt work in the client on amazon fire tv 4k max that has av1 this check "Build.VERSION.SDK_INT >= Build.VERSION_CODES.Q" in jellyfin/androidtv/util/profile/MediaCodecCapabilitiesTest.kt will fail since the stick reports android 9

adb shell getprop ro.build.version.sdk 28

adb shell getprop ro.build.version.release 9

[Yeetov]

Is this the same code for the fire stick (firetv) or just android TV?