openhab-core
openhab-core copied to clipboard
openhab
New feature media to simplify handling of media in Openhab
Hello,
The idea behing this is to try to unify and simplify media handling in Openhab between differents addons. We already have in openhab today:
- A PlayerItem that enable to controls a player expose by an addons.
- An audio package that give some level of abstraction on Audio Input and Output source, enable recording and playback to different device.
That would be great is to have a new media package that will abstract the Media calalog, enable to browse this catalog in unify way, and handle the playback through the differents addons.
Notes that there is already some support for this today, but handle separatly in each addons:
-
UPNPControl addon is able to browse media collection using a text input that is populated with song from the media library : this work, but very hard to use because of poor UI.
-
Spotify addon is able to show playlist, and start playback of them. It use like UPNP a text box populated with playlist.
What is done today:
-
Add a core package org.openhab.core.media that implements base interface, and enable each addon to register to expose it media collections to a central mediaRegistry.
-
Same package expose a Servlet that enable browsing of this mediaRegistry.
-
Add a package org.openhab.core.io.rest.media that expose the results on the reset layer.
-
Add some pages in UI : pages/media.vue and pages/media-popup.vue that enables browsing of the mediaRegistry.
-
Modify Spotify and UPNPControl addons to handle this new media Stuff.
-
Some modification on PlayerItem control to launch the media-popup to enable the browsing, and send back the result on the PlayerItem channel, that will enable to start playback through the respective addons.
How it currently locks:
On the mediaPlayer, a new Icons allow to launch the popup.
Popup will display the different MediaSource, so you can browse down your media collection:
When you go down to Track levels, it will show you a screen displaying the parent Album / Playlist, and all the tracks inside, with A global Play button, and the possibility to click on each track to play them individually.
What is not working / will need to be done: A lot of thing, this is very preliminar version. So I will not do the list tonight. The idea is just to open the discussion to see if it's a good idea or not.
The Pull-request involved: https://github.com/openhab/openhab-core/pull/4709 https://github.com/openhab/openhab-addons/pull/18501 https://github.com/openhab/openhab-webui/pull/3140
This issue has been mentioned on openHAB Community. There might be relevant details there:
https://community.openhab.org/t/ideas-and-discussion-what-features-do-you-want-in-openhab-5-0/160573/709
I will join Dan here, amazing feature! Looking forward to review your UI PR.
Hello @All,
@florian-h05, @digitaldan, @lsiepel,
I would apreciate if someone with more experience in core that me can give comments on this first version. Especially, about the PlayerItem modifications.
I finally decide to drop my new BrowerMediaItem implementation, and modify the existing PlayerItem. I introduce a new MediaType accepted command on it. The idea is that this MediaType command would be use to play / queue an album / track / artist to play on the PlayerItem.
The MediaType is a complex type compose of:
- The type of command : PLAY/QUEUE (for now).
- A parameter (that id the id/path of entry to play).
This command would be relay from core to addons, and addons would have to implement if to handle the play/queue request. I've currently test this approach with spotify and upnp addons, and it seems to work (minor some obvious bugs to handle).
Laurent
Can you share your build/zip of the openhab folder with these new features included? I would like to start working on adding media handling to some of my bindings.
Hi, I wanted to know about the progress of these functions, it is very interesting to see the functionality
Hi, I wanted to know about the progress of these functions, it is very interesting to see the functionality
I've restart to work on this last week, and make some great progress. The UI is now more stable. I've also add a way to select the output device directly from the control player.
But of course, there still a lot of work to be done. This will clearly not be ready for 5.0 release. Perhaps for 5.1 release if we find time this summer to work on it, or a worst in 5.2.
re your build/zip of the openhab folder with these new features included? I would like to start working on adding media handling to some of my bindings.
Sorry for my last response. This is not very easy to share build/zip of core feature. If you want, I can help you to setup a dev environment with the new feature. If you don't want to go this way, another way will be to grab a build artefact from github actions.
Currently last build is KO because of a failed unit test, I will try to fix it soon so we have build artefact.
Can I ask you on which binding you would like to add this feature ?
Laurent.
Hello,
This is a quick list (not complete) of what I see need to be done before reviewing this new features for inclusion in 5.1 or 5.2 :
- [Upnp] : service handling is broken (Emby).
- [Upnp] : sometimes seems to mix entries (if slow response from UPNP server).
- [Heos] : support (in progress)
- [Heos] : output device not initialized correctly every time.
- [Heos] : support selecting output device.
- [Squeezebox] : Implements Lyrion/Squeezebox media browsing and playback.
- [Spotify] : Implements the paging functionnality.
- Verify that paging request are implements in all binding to be sure to have all entries in media-browser.
- Verify that play / enque is working ok with all implemented bindings.
- Verify that all commands working ok with new MediaType command.
- Review the proxy feature to enable specifics setup configurations.
- Filter Browse source when coming from specific player attach to a specific binding.
- Add a default media control player to /media menu page (enable quick access to playing a media).
- Enhance UI and fixed bugs.
- Implements the searchbar functionnality.
- Implements the global player functionnality.
- Verify that all entries are correclty populated on all bindings, today some are empties !
- Verify that imageArt are correctly handled everywhere, today missing at some places.
- Review icons for media-browser, some have to be fixed.
- See if we add video support, currently only test with Music source, but should work also with Video as weel.
- Divide MediaType class to 2 separates class to differentiates between ACCEPTED_DATA & ACCEPTED_COMMAND and make things more clear.
- Do we need to show deviceSelector if we are bind to specific control player ?
- Some way to see the media queue directly from openhab, and perhaps have some action on it ?
- Fix unit test in openhab.core to handle more possible case.
- Have some people test this new feature to have some feedback and fix bugs.
- Add support for other bindings.
- Make sure of backward compatibility for binding not implementing the new feature.
- Code cleanup and static analysis review.
- Other ?
Done:
- Backport proxy feature in media Servlet to openhab UI so we can see cover art ever if we are not on inside network.
- See if we can find a better way to display some path in navigator, today some technical path like "spotify:playlist:7icTs90EG8GG6tGEDnATdi", not very great.
Hello all, @digitaldan @florian-h05,
I go back on working on this feature, and make some great progress last week (better UI, handle of Heos, ...) But still have a lot of things to be done.
Has it is a little hard to finish because it touch on many parts of openhab : core, bindings, UI. I was asking myself if we can think of adding a feature flag to deliver it disabled in next openhab, but that people interested in testing it, or updating there binding will have some way to enable it in simple way on their environment.
I don't know if something like that have already been done before in openhab. I think it could be simple as the feature in core have not a lot of dependencies, so it can be toggle off easily.
And UI and binding parts can look at the state of the feature in core to only enable code mods if feature is enable. Let me know if it can be an option, if so I will look how we can implement this.
Laurent.
A few screenshoot of latest changes:
- Add a media menu in Main menu to have a quick access to the media-browser.
- Support new binding types like Heos and Squeezebox.
-
New rest endpoint to browse available output Devices :
-
Heos support will enable to handle external services like Tidal, Soundcloud, Tunein
- Sample Heos / Tidal browsing
- New Device Selector feature directly on MediaControl item
Attach the current bundles for core modifications and the modified jar bindings.
I've also update my test server for demo purpose : https://openhab-dev.clae.net/ Fill free to have a look, but just don't hit the play command as it is links to my real device at home !
Have some new features this last days
- Most binding will now support paged browsing : test with Spotify, Upnp, and Heos.
- UI implements infinite scrooling so page results are automatically load when you scroll down in the browser.
- Add a searchbar function on top of media browser : today only test with spotify plugins.
- Start added a global media menu and a global player capabilities.
- Review the icons.
This issue has been mentioned on openHAB Community. There might be relevant details there:
https://community.openhab.org/t/new-feature-media-to-simplify-handling-of-media-in-openhab/164707/1
Hello,
A little update about the progress on feature-media branch. I've made progress on it last week, fixing a lot of issues on the UI side. I've also implements some functionalities that were missing on bindings.
For memory, binding that currently support the feature-media are : Heos, Spotify, Squeezebox, Tidal and Upnp. Tidal is a new binding, and will be available only after the release of the feature-media branch as it depends on change of this branch.
New functionality that was push this week are:
- Support for a Media Queue functionality that enable to see was is currently playing.
- Add support on Squeezebox binding to browse media collections (previously only support favorites songs).
- Tidal now are able to browse the media collection (Album/Artist/Playlist/Track), and basic but not really working (not always) playback to Upnp renderer.
- Basic support to cross binding playing : if a currently selected player not support direct rendering, mediaService can ask the source binding for a stream, and render it on the target player.
- Hoes binding enhancement to display better images on sources.
- Various fixes on UI : device selector more stable, global player fixes, and some mores.
There still a lot of thing to be done before releasing. The most important issue by priority are:
- Check that the UI is compatible, and upgrade if necessary with vue.js v3 branch : #3350.
- Fixed the Search functionality that is broken in latest release.
- Finish the implementation of Media Queue.
- Fixes the different bug that I've found so far.
I'm still asking myself about simple question on how to release this has it is a lot of modifications, both in core, bundles and UI.
Perhaps a good way will be to release first the openhab-core modifications. The modifications have not impact at all on other openhab features until they are used by bindings or UI. So perhaps we can integrate it at first, even if not fully finished / full features.
The next one could be the UI part : This is a little more difficult has we have to make sure that there is not regression with previous version. I will need to check that new functionalities will be enable only if mediaService is present in core. I will also need to check that component like oh-player-control work ok with old code.
This will then enable binding maintainers to test they respective bindings, and comments about possible issue on thems.
To end, a few Screenshot about last modifications:
- Global player is now more stable. Displaying current playing song, track progress, title, artist name.
You can change global player target device using the device selector:
After changing, the global player will reflect the state of the new target devices
- Player as a new media queue icons
When you click it, UI will display a new Media Queue page, with the current playing list, and the possibility to see what song is currently playing, and change it.
- New entry in Media Browser for Tidal plugin, with capacity to browse the tidal collections:
- New more readable icons on Heos bindings
Some little news,
-UI: The UI part was ported to vue3/Framework7 v7/Pinia store. Most of the work done (about 90%), but still a lot of test to be done to verify regression and fix possible issues.
-
Tidal: Work in progress to support HI_RES_LOSSLESS quality (up to 192khz/24bit). Most of the modification to be done was identified, but still need to work on it for implementations.
-
Sonos: Sonos binding will be certainly next biding to receive support for feature-media support.
Appart this, no more change for this week !
Laurent.
I just discovered this, and there is a lot in here, but I'd say that generally this would be a great feature, but I suspect that it can also be an enormous task to get it to work properly across the board. Anyway, great initiative 👍
I have a few (somewhat random) comments from things I picked up when browsing through the thread:
- Player as a new media queue icons
Having buttons to seek is great, even if it can be difficult to make useful because of the general "laggyness" of working through all the different layers. A "slider" is generally more suitable for seeking with high latency UIs. But, having a button is still good for those situations where a slider doesn't fit/can't be controlled, even if it will have to just skip some preprogrammed "step" with each click.
What I see is that there's really a need for a "Stop" button as well. Both upnpcontrol and the chromecast binding have separate switches today for handling "stop", it's not really elegant, both because it's separated from the other controls, and because it's a toggle switch. Maybe it would be possible to make pause/stop a combined button, where long-click or double-click functioned as stop?
Another thing that I've learned from working with UPnP, is that there should be a way to enable/disable the different buttons. Some media support seeking while others don't for example (live broadcast or transcoded media). UPnP:AV has variables that reflect which "play actions" that are supported, so that the renderer can inform the controller of this. That would disable the seek buttons when seeking isn't supported, or disable prev/next when that's not a thing (just playing a single media).
Regarding the name "MediaType" mentioned above, I would caution against using that term. It's a term that can mean so many things, that it's just generally confusing. They renamed MIME types to "media types", but most people still call them MIME types. I believe the main reason is that even though "MIME" itself is misleading, you at least know that you refer to a "text encoded in a particular way to identify a type of file or media". Media type can also mean for example whether it is audio, video or images. I would try to avoid the term if at all possible.
I recently looked at the AudioFormat class, and I discovered that it has some major flaws. I don't know if you've looked at that, but to make media playback work somewhat reliably, you need to have a way to convey what is and isn't supported. The AudioFormat largely fails at that for audio files, both because the distinction between file-format, container and codec isn't clear (and it can be difficult to draw this line sometimes), but first and foremost because it doesn't handle ranges for the properties. Take sampling rate for example - a typical player can perhaps play 22.05, 44.1 and 48 KHz, one or two channels. Some can even play some sample frequencies for some number of channgels, and a more restricted range for a different numer of channels, like:
22.05, 44.1, 48, 96 for <= 2 channels, but only 44.1 and 22.05 for 3-5 channels.
To be able to actually match up sink and source media in a meaningful way, you must be able to "match" these things. Currently, AudioFormat allows you to only specify one value, e.g. 44.1 KHz or null/any. Restricting it to only one value will be way too restrictive in most situations, so people will just have to use null/any for all the properties - and you have effectively disabled any "matching" at all.
It should also be noted that MIME/Media type are completely incapable of conveying compatibility information at all. They mostly just indicate the file format, often just vaguely. In addition, there is so little consistency in how they are used, that they are close to useless (for media files). It's sad, but that's the truth of the matter, and it's probably the main reason for the "failure" of UPnP:AV and why DLNA came to be in the first place. DLNA's way of "handling" this is to define a huge number of "media profiles", with very specific requirements. Players will then have to inform which media profiles they can play, and servers will have to figure out what profiles a certain media file qualifies for. That's not always easy, because they go into extreme details like how large the decoding buffer should be or how Huffman tables are organized. Regardless, it's much better than trying to make things work based on MIME/Media types.
Good evening @Nadahar,
Thanks a lot for your message. You are one of the first to get some feedbacks on this functionality other that it will be great. I need comment like yours to progress on the implementation. And I'm sure your comment will be helpful to me !
Laurent.
Hi Laurent, great to watch your progress!
I'm still asking myself about simple question on how to release this has it is a lot of modifications, both in core, bundles and UI.
Do core first, then UI and add-ons can be done in parallel, though having some add-ons implementing it will be helpful for UI testing.
I will also need to check that component like oh-player-control work ok with old code.
Once the backend parts are in place I can also help testing. The UI screenshot look pretty nice already.
What I see is that there's really a need for a "Stop" button as well. Both upnpcontrol and the chromecast binding have separate switches today for handling "stop", it's not really elegant, both because it's separated from the other controls, and because it's a toggle switch. Maybe it would be possible to make pause/stop a combined button, where long-click or double-click functioned as stop?
Good point, and I like the idea of combining it, as we have to keep in mind that the UI is also used on phones, where you can't put a hundred buttons on the screen.
Another thing that I've learned from working with UPnP, is that there should be a way to enable/disable the different buttons. Some media support seeking while others don't for example (live broadcast or transcoded media). UPnP:AV has variables that reflect which "play actions" that are supported, so that the renderer can inform the controller of this. That would disable the seek buttons when seeking isn't supported, or disable prev/next when that's not a thing (just playing a single media).
How would we get that information from the binding to the UI? I can imagine fitting it somehow in the state description, as this can aleady be provided by a binding, and is available in the REST API. But I am not sure what exactly you can put there.
Note that I've extended the state description in regards that we have before. It can now be a simple state as in old version, or a compound JSon object that take lot more information has you can see bellow.
So if we need more information go to the UI from binding, we can just add it there !
But this is the actual Item state? If so, keep in mind that all this is transmitted over SSE/WS on every change. Static information like capabilities should rather be put in the Item's state description so it is retrieved via REST and not sent as Item state.
I am generally wondering if there is a better way to get all that "live" information around (but haven't found one yet), especially modifying the Player Item in a breaking way where the default state type changes is giving me some headaches.
How would we get that information from the binding to the UI? I can imagine fitting it somehow in the state description, as this can aleady be provided by a binding, and is available in the REST API. But I am not sure what exactly you can put there.
I honestly haven't thought of/considered that part of the equation. I guess there could be separate (boolean, read-only) channels. Remember that this can change rapidly, from media to media (some supports seek, others don't, the first in the playlist has no "prev", the last in the playlist has no "next"). Can state description handle this rapid change? The channels could be called e.g "canSeek", "canNext", "canPause", and if they didn't exist, all the buttons would always be enabled. It would obviously be preferable if you didn't have to use dedicated channels for this, but as I understand it, that's the mechanism that can support "frequent changes"?
The buttons could be hidden instead of visually "grayed out" when disabled, but it means that there could be situations where the media player bar could be somewhat "asymmetric", so it's probably better to gray them out..?
edit: Ah, I think I understand now, these commands aren't separate actions today, they are all packed into one channel? That would mean that this information would need to go in there too, yes. This "combined" design might have some limitations, especially in wasting bandwidth in repeating the same information over and over again. The bandwidth part could be compensated by using bitwise or to combine them all into one integer value. You can support 32 boolean states with one integer, except that maybe you can't? Because AFAICR, JavaScript doesn't have an integer type and probably can't deal with this? Anyway, with the number of buttons actually in use here, it might be possible? A quick estimate is that those that would need enable/disable would be "prev", "next", "seek", "pause" (not everything can be paused) and maybe "playlist"? This only requires 5 bits, so it can be "encoded" in the values 0-31. That should be well within the capabilities of JS. But, doing this to save bandwidth would of course mean a bit more work in both ends, encoding and decoding the information.
Can state description handle this rapid change?
No, they aren't meant for this. (There also was a discussion some time ago about (ab)using metadata for storing dynamic additional data, and there the consensus was that Items should be used for storing highly dynamic data.) There is also caching in between on the REST layer ...
The buttons could be hidden instead of visually "grayed out" when disabled, but it means that there could be situations where the media player bar could be somewhat "asymmetric", so it's probably better to gray them out..?
With some CSS it should be possible to have everything aligned properly automatically.
Because AFAICR, JavaScript doesn't have an integer type and probably can't deal with this?
It does: BigInt. I have a use case where I receive parts of a string as int from ModBus, then interpret the Int as a series of chars. See https://github.com/smartheit/openhab-js-tools/blob/76b450fd5d68569b85c32638d51454c377e62c50/src/utils.js#L15.
But, doing this to save bandwidth would of course mean a bit more work in both ends, encoding and decoding the information.
You are right, but I am no fan of having encoded data in Item states. Most users expect to be able to look at an Item state and see its value, encoding data contradicts this.
Currently, everything is squeezed into the MediaStateType on the PlayerItem. IMO this is a bit too much for a single Item type.
If I had a better understanding of what data is coming from where, and needed for what, I could try proposing something better.
With some CSS it should be possible to have everything aligned properly automatically.
Yes, I was thinking of "asymmetric" in the sense that the play/pause button might not be "in the middle". I don't know if that would matter though.
Currently, everything is squeezed into the
MediaStateTypeon thePlayerItem. IMO this is a bit too much for a single Item type. If I had a better understanding of what data is coming from where, and needed for what, I could try proposing something better.
I'm afraid I don't have this overview either - I just see that there's a need to communicate this, to avoid having buttons "that don't work" with no way for the user to know when they work and not.
I'm afraid I don't have this overview either - I just see that there's a need to communicate this, to avoid having buttons "that don't work" with no way for the user to know when they work and not.
My concerns is mainly not about transmitting the information which "capabilities" are available, but rather the example Item state from https://github.com/openhab/openhab-core/issues/4710#issuecomment-3506539176.
My concerns is mainly not about transmitting the information which "capabilities" are available, but rather the example Item state from #4710 (comment).
Yes, this might be a good time to stop and think about how the design should be. It's almost like trying to turn a channel into a "group of channels", and then sneak a lot of information through in ways that probably will cause problems down the line.
Could the whole "media playback widget" be based on a group of channels instead of a single channel? I don't know if this is even feasible, but it feels like something like this whole scenario would fit better there than stuffing everything into a single channel.
s, this might be a good time to stop and think about how the design should be. It's almost like trying to turn a channel into a "group of channels", and then sneak a lot of information through in ways that probably will cause problems down the line.
Could the whole "media playback widget" be based on a group of channels instead of a single channel? I don't know if this is even feasible, but it feels like something like this whole scenario would fit better there than stuffing everything into a single chan
Have not respond to the few last messages as I'm busy on other stuff this week. Just to give you a little background about the modification I've made about MediaStateType. My concerns was how to handle multiple information on MediaPlayer without linking multiple channels to the PlayerItem.
The idea of Nadahar about "group of channels" is interesting. But It not exist today in Openhab, and would need a lot of modifications in openhab.
At time I've start feature-media branch, player control was mainly based on PlayPauseType, NextPreviousType. So there was very little feedbacks informations on player control. You was also able to bind the player control to an item for current Artist and current Playing Track. But any each information you need would need to have another channel bind to the item.
The other issue I've encountered was that I want to introduce a global Player control in the UI, and that this player control can toogle between the different available Player control present inside Openhab just by selecting one inside a popup controls.
So the most easy way I've find to solve this issue was to add a new ComplexType (MediaStateType) attached to the player control. The idea was to have the dynamic data directly in this type encode in a Json, and try to make it not to heavy so it can fit the SSE update system.
This work quite well, but just need:
- For the UI to support at the same time the old PlayPause adn NextPreviousType as well as the new MediaState so we keep it compatible until new system is implements everywhere.
- That each binding start to implements the return of new MediaStateType in addition of Old types.
And it also work quite well on having loosely coupling PlayerControl that you can switch from one PlayerControl channel to another one using the device selector popup.
In general, it might be worth the time to ponder thoroughly about such design issues. Even if something works nicely when you test it, it might not work as nicely when in a production system, where a lot of other things are going on. Things also have a tendency to evolve, so that more functionality is added, making a load that might be tolerable initially untolerable. In general, it's important not to keep transmitting redundant data. I don't know the details of this solution, but if it keeps repeating all information if just one has changed, or if none has changed, I'd say that is a "design problem" regardless of whether or not any problem can be observed when testing.