architecture
architecture copied to clipboard
home-assistant
Decouple integration releases from core
This is a segway from #192.
I want to explore if it is possible to decouple integration releases from core releases. The idea would be that a bug fix or new feature added to an integration can be released outside of the normal release cycle.
This issue is mainly trying to list all the things that we have to think of, and explore possible solutions.
Semver Home Assistant core
We should start releasing Home Assistant Core using semantic versioning. Major upgrade means a breaking change, minor means new feature, patch means bug fix. This applies to all core files, helpers and utils.
Manifest.json to add supported core version
Integrations should specify in their manifest the min and max versions of Home Assistant Core they support. This allows us to tell in advance how an impact of HA Core or an integration will impact the system.
If you update HA, the following integrations will automatically be disabled because they are no longer compatible: X, Y, Z.
Proposal #195 will help with this.
Integration storage
Integrations are going to be able to be updated. We can decide to no longer bundle any of them with HA, or still bundle them, but allow them to be overridden via updates.
The only downside for Hass.io / Docker is that they don't contain build tools. So updating integrations that require compilation will need to be able to access pre-build wheels.
Integration updates
Updates of an integration won't override the existing integration on disk, but instead be installed in an adjacent folder. This will allow us to unload the config entry in the old integration and load it in the new integration, rolling back if set up fails.
This requires integrations to only do imports locally (from . import X), or use hass.components.X to refer to other integrations. Importing from HA Core should be fine.
Having integrations live on different paths is already supported by the loader.Integration class in Home Assistant.
# This can load from anywhere
integration = await loader.async_get_integration('hue')
component = integration.get_component()
platform = integration.get_platform('light')
So this allows us to have 2 versions live side by side. This won't work if an integration is a dependency of another integration, in that case we should ask the user to restart.
Core API documentation
We need clearly define the core API, and better document. Otherwise, integration developer will hard to know the core change and which version is compatible for their component.
Having a fall-back mechanism would be really a good thing.
If I think about it bit further then this could mean that we would be able to catch issues in the dependencies which renders the actual integration unusable after an upgrade or even avoid breaking of the integration after changes between releases with a rollback option.
We should start releasing Home Assistant Core using semantic versioning. Major upgrade means a breaking change, minor means new feature, patch means bug fix. This applies to all core files, helpers and utils.
hallelujah!!
Integrations should specify in their manifest the min and max versions of Home Assistant Core they support.
Lets skip the max version. The integration versions should be aligned with the core version. IE: Core 2.x only uses integrations 2.x. Then there can be integration 2.x-3123.2 if they like, but specifying a max version will only be a pain in the *** as that will need to be updater for every core release. Or I might have misunderstand something here.
This requires integrations to only do imports locally (from . import X), or use hass.components.X to refer to other integrations. Importing from HA Core should be fine.
I don't see this as a problem. This is normal development practice.
What about breaking this down even further? You could run home assistant in a microservice architecture where each integration lives in a separate docker container and talk to a core which only takes care of authentication, event handling and persistence and automations which could also run in appdaemon or nodered.
We could even run this in kubernetes cluster on top of k3s or in a haasio way.
This also would allow integrations to crash or slow down without hurting the core or other integrations.
It would also allow to scale the system. You could just add a bunch of nodes to a cluster or to have a fault tolerant solution.
I don't think we should force containers, however we could make it possible, but again, not mandatory.
If we would have a small seperate process per integrations users could still run it without containers.
Maybe home assistant could use mqtt for the connection between the core and the integrations. This would also allow developers to decide which language they use. Imagine each integration would just work like zigbee2mqtt...
@runningman84 I don't think your suggestion is the way HA would follow. The beauty of HA is the asynchronized super fast core and integrations. Move the core-integration communication to inter-process, inter-container, even over-network will transform HA into a different project.
The main issue I can see with doing this is going to be managing requirements. We'd have to check that any requirements specified in the manifest by more than one component are the same version, and probably have a process where any shared requirements are pinned across the same core version.
This probably means having a "supported requirements" list for a particular hass core version.
I agree that containerised integrations are going to be over-kill, but they do solve the requirements issue.
Surprisingly, up to this day, conflicting requirements have been a very small issue for us.
Suggest to checkout the Add-ons architecture/structure concept from the Kodi (formerly XBMC) project.
As you can maybe gasp by their repository tree structure as linked in the different repositories below having a different tree of addons for each major release of the Kodi core allow them to keep different addon components versions for different major versions of the Kodi core. That way addon components Kodi 18.x (Krypton) major release does not have to be on the same release schedule as addon components Kodi 19.x (Leia) major release.
A nice blog post from Kodi on the good, the bad, and the ugly sides to supporting unofficial repositories:
https://kodi.tv/article/repos-when-all-one-can-be-no-fun
This is how the XBMC Foundation has solved its decoupling of integration releases from its core releases:
https://kodi.wiki/view/Add-on_manager https://kodi.wiki/view/Add-on_development https://kodi.wiki/view/Add-on_structure https://kodi.wiki/view/About_Add-ons https://kodi.wiki/view/Script_sources https://kodi.wiki/view/Special_protocol
https://kodi.tv/addons
https://kodi.wiki/view/Official_add-on_repository
https://kodi.wiki/view/Unofficial_add-on_repositories
https://github.com/xbmc/repo-scripts https://github.com/xbmc/repo-scripts/tree/leia https://github.com/xbmc/repo-scripts/tree/krypton https://github.com/xbmc/repo-plugins https://github.com/xbmc/repo-plugins/tree/leia https://github.com/xbmc/repo-plugins/tree/krypton https://github.com/xbmc/repo-skins https://github.com/xbmc/repo-skins/tree/leia https://github.com/xbmc/repo-skins/tree/krypton https://github.com/xbmc/repo-webinterfaces https://github.com/xbmc/repo-webinterfaces/tree/leia https://github.com/xbmc/repo-webinterfaces/tree/krypton https://github.com/xbmc/translations https://github.com/xbmc/translations/tree/leia https://github.com/xbmc/translations/tree/krypton https://github.com/xbmc/repo-resources https://github.com/xbmc/repo-resources/tree/leia https://github.com/xbmc/repo-resources/tree/krypton https://github.com/xbmc/repo-scrapers https://github.com/xbmc/repo-scrapers/tree/leia https://github.com/xbmc/repo-scrapers/tree/krypton
Addon components can be skins, web interfaces, translations, shared resources (such as example sound files or weather forecast icon images that multiple skins and scripts/plugins can utilize), scripts as content addons, scripts as inline plugins, scripts as modules only uses as dependencies by other scripts (and often even shared between different addons), scripts as background services, web scrapers, as well as binary addons such as codecs, PVR clients, games and game emulators, screensavers, visualizers, and many more:
https://kodi.tv/addons#categories
All these types of add-on components for Kodi can have separate releases from the core application that is Kodi. FYI, originally Kodi only supported Python based addons, but now it also supports binary addons and shared image resources too.
Looking at the different addon types there, it's worth contemplating how lovelace addons and custom cards etc will fit in to the greater scheme of things. After all, a custom component is pretty likely to have its own custom state card.
@iantrich yes of course, what I'm implying here is that ideally it would be good if they could be packaged the same way as components and downloaded from an add-on site much like https://kodi.tv/addons . After all end users don't care if it's a front-end or back-end addon, they just want the functionality.
We already offer it partially: Integrations can already bundle and register their own panels and HTTP endpoints.
This thread should only discuss how to decouple first two rows.
| Category | Criteria | Support (Source Code Repo) |
|---|---|---|
| Core | N/A | Home-Assistant organization |
| Built-in / Native / Essential Integration | Hight quality, meet all of our requirement | Home-Assistant organization |
| Custom integration (Custom Components, Custom Panel, Custom Card, etc) | Might not meet our design principle or requirement | Community |
FYI, Kodi's https://kodi.tv/addons pages did not use to have a "download" option, as those pages previously were only a browser with information about addons and links to wiki or the code. Before this, if a user wanted to download an addon they either had to use Kodi's Add-on Manager which is integrated into Kodi's core and part of its GUI (which in IMHO is still the best user experience for installing addons in Kodi
https://kodi.wiki/view/Add-on_manager
Kodi's Add-on Manager tries to emulate the feeling of a professional app-stores such as Apple App Store or Google Play Store). Point is that do not focus to much on the https://kodi.tv/addons pages as that is not a core feature of Kodi's Add-ons architecture/structure concept, (the web pages are only nice to have, not need to have).
@awarecan FYI, Kodi has already gone through similar processes of decoupling all of its built-in integrations and made them into add-ons components which today allow them to implement "official essential integrations" the exact way as the "communities custom integrations".
https://kodi.wiki/view/Official_add-on_repository
Kodi now has several official Kodi addons that developed and maintained by Team Kodi, but the APIs and addon implementation they use is still the same as unofficial third-party addons from the community.
https://kodi.wiki/view/Unofficial_add-on_repositories
All of Kodi's addons are decoupled the same way if they are of the same type. So from an Add-ons architecture/structure concept point-of-view of the implementation, there is no difference between "official essential integrations" and "communities custom integrations".
@Hedda please see my previous comment, because HA's design, the event loop and state machine is very easy to be abuse if you have a bad actor installed. So we have to do thorough code review if we accept an integration into our code base. The "official supported" integration and custom integration has big different in terms of average quality. And also in most part, custom component has no different then built in component. We are working on filling the gap in between.
Anyway, I hope we focus on discuss how and what need to decouple core and integration, not continue on what is integration.
@awarecan I understand that and all I am saying is that if you take a closer look then you will see that Home Assistant is not unique to have encountered these issues and is in fact no different from Kodi in this case either, as Kodi has the exact same dilemma (or worse as they have been over the year got a lot of bad press because of people having difficulty understanding the difference of Kodi officially supported addons, approved community addons, and unofficial third-party add-ons which way in some cases even be banned or blacklisted addons).
- https://kodi.wiki/view/Official_add-on_repository
- https://kodi.wiki/view/Unofficial_add-on_repositories
- https://kodi.wiki/view/Official:Forum_rules/Banned_add-ons
Kodi addons also have to go through review before they are approved to be added to the official Kodi addon repository, many submitted addons are not approved and thus does not get added to the official Kodi addon repository. That does not change that official and unofficial addons use the same APIs and addon structure from an architecture point-of-view.
A nice blog post from Kodi on the good, the bad, and the ugly sides to supporting unofficial repositories:
- https://kodi.tv/article/repos-when-all-one-can-be-no-fun
If you would reach out to the Kodi developers and ask them to summarize their journey of decoupling and history why their addon architecture looks the way it does today then I think you would find them very accommodating, and I would even guess that many Kodi developers are also users of Home Assistant themselves and would be more than happy to explain the pros and cons of their choices.
Anyway, if you do not want to look at Kodi then that is fine, sad but fine, it is your loss. I will stop my comments regarding Kodi's addons concept now.
Kodi uses the same terminology as Linux distros with repositories. Perhaps we could adopt a similar scheme, with a list of approved integrations and then users can create their own lists, which could be added in the future as an install dialog.
A very similar method I have been using recently is with OctoPrint, which has its own repository of plugins which can be installed from the UI (This uses pip to install these plugins and shows the output). They also have the option of installing plugins from a URL (with the warning that these are unverified).
A repository / integration management system sounds like a good idea to me. Obviously, this is a way off, but if the plan is to let vendors create and manage their own integrations, I think this would be a recognised and good plan of action 😺
I would rather see a pip setup plug-in endpoint. So we expect integrations to be pip installable. We define a location for where they are allowed to install. Simar to how pytest handles extensions. Why reinvent the wheel with hosting a repository?
https://packaging.python.org/guides/creating-and-discovering-plugins/
Want to recommend reading this new blog post from the Kodi/XBMC developers regarding the good, the bad, and the ugly when it comes to having options available for end-users to install add-on components from both Kodi's "official" and third-party "unofficial" sources (a.k.a. "repos" when it comes to Kodi) :
https://kodi.tv/article/repos-when-all-one-can-be-no-fun
This blog is written by Kodi/XBMC developers and described from both a developers viewpoint with security in mind and managing malware risks as well as from the viewpoint as end-users too. So this blog post goes also covers what Kodi developers can do to make the user experience better,
Having the components in the same repository as Home Assistant has following downsides:
- A component cant be updated without updating everything
- A component does not have its own issue tracker, everything is in home-assistant/home-assistant
- A component cant be forked and developed further by the community
A better approach, in my opinion, would be moving each component in its own repository. Also each repository should be installed and updated by a package manager (e.g. npm). Home Assistant might be released containing no components at all but download them on first installation.
The guys from Angular are doing a fantastic job on that using npm to require, handle and update tons of in-house-modules/components and tons of third-party-modules with all their different versions just by use of npm.
The HA team is steadily implementing features that will make this viable, the features that are getting added to manifest.json are the biggest demonstration of this. I don't think we need persuading that third party packages are a good idea.... I think the biggest blocker is really we just need to have a stable set of core API's, otherwise this will just increase the fragility of releases.
To play devils advocate a bit I think the linux kernel is the best counter example. It has the same downsides and the bar for contribution is high. But in tree drivers are better maintained and of higher quality because of the extra eyes on them. Third party out of tree wifi drivers tend to make everyone sad. They might only work with old kernels, or they might just crash a lot.
The current approach is not perfect, but speaking as a recent contributor to HA (i work on homekit_controller) I hope my component continues to benefit from being "in tree":
- When there are changes to core API's or components that affect my component often the person driving that change will patch my integration, and it will get shipped in a release, even if i am busy in the real world and dropped the ball.
- The tests for my component get run against every HA commit, so if a core change regresses my integration everyone finds out before a release.
- When i was a "fresh" contributor I was maybe a bit sure of myself and frustrated that code I had "finished" in january wasnt merged yet and it was march. I was frustrated that "obvious" changes were "too big" and needed breaking down into pieces so they could be reviewed. The reality was my code wasn't finished, and it took much longer than that to get it right. My work on HA would not have been as good or as reliable without code review from HA code devs.
These things are still possible in other models, but they will have extra friction and happen less frequently or less well.
Ultimately, I want my code to be where it is now, I want the benefit of the experienced HA devs watching over me, and I want my code to be used with the version of HA it was tested with and only that version.
With this sort of thing I think there aren't solutions but rather you have to choose where you want to experience the pain. And there is always pain with package management.
A better approach, in my opinion, would be moving each component in its own repository.
While in principle I've been on the same side of that argument, there's a massive problem with it: Having a separate repo for each integration is a huge overhead.
Currently, there are ~800 "integrations" (homeassistant/components folder). Creating a new repo, setting up everything (CI, bots etc) would even with a lot of automation prove to be a monster issue. Plus especially since you're mentioning issue trackers: Many integrations in HA have no active developer, and spreading everything over 1000 github links will make maintaining much more difficult. Same for code review. I like being able to visit one link and see all issues/PRs. We have labels to categorize issues.
Then there's the issue with checking out all these repos. What if a core dev needs to find(+replace) across all integrations? Practically impossible with multi-repo setup.
Next, how would you even create a PR for a new integration? Because there's no repo to create it against, how would you create a PR? Yes submodules can solve some of this, but that would put the repos out of HA's hand, and if the dev is no longer responsive it needs to be forked etc.
I think this is a dead end. For non-official integrations that may be a solution, but we can't change to such a model now.
One way to decouple core releases and integration releases could be to split the repo into 2 repos:
-
homeassistant/components(the integration repo) - Everything else (the core repo)
Both repos would include the other in their CI tests. Changes to the core would test using the latest release of the integrations repo, and changes to the integration repo would test using the head of the core repo.
If a change in the core requires changes in the integration repo, that developer will have to make the necessary changes to the integration repo, and the integrations would have a special release that keeps them compatible with the core. Users will not be able to update their integrations to this release or later until they update their HA core.
I would also think decoupling/breaking-out all integration component to enable separate releases from Home Assistant Core should probably be a prerequisite for allowing the possibility of having LTS (Long-term support) support versions of Home Assistant Core?
https://en.wikipedia.org/wiki/Long-term_support
APIs for third-party services break all the time, so without decoupled integrations you can not really have a 'no-touch' LTS release.
@Gamester17 Depends on the definition. For 1.0 is has been determined that we won't be splitting the core integrations. 1.0 would be about the stability of the internal APIs of Home Assistant itself, not its integrations. However, who knows what the future brings.
@Gamester17 Depends on the definition. For 1.0 is has been determined that we won't be splitting the core integrations. 1.0 would be about the stability of the internal APIs of Home Assistant itself, not its integrations.
Personally I think a Long-term support release does not have to be a Major release, it's not even the norm, the Linux kernel official long-term releases are for example version 4.4 and 5.4. So I didn't assume HA 1.0 would become an LTS release when I mentioned it.
Just suggesting that splitting components with integrations to third-parties from core is probably a good idea before an LTS release.