[D8] Allow enabling modules (and themes/layouts?) via config sync / import

[klonos]

We have discussed this possibility during this week's dev meeting (https://youtu.be/s0Jsh1_lIjw?t=2491), and have agreed that at the very least it would be possible to enable modules from config (this is less risky than disabling/uninstalling modules, which has the risk to lead to data loss).

In Drupal 8+, this is achieved via the core.extension.yml config file. Perhaps we should similarly introduce a core.extensions.json file, or system.extensions.json, or reuse the existing system.core.json file for that.

[klonos]

Noting that layouts (layout templates that is) are always "enabled" (rather "available"), unless excluded via layout.settings.json. So we already provide a way for people to "enable"/"disable" them via config.

[klonos]

I've created a PoC PR here: https://github.com/backdrop/backdrop/pull/3692

• an update hook creates a system.extensions.json config file*, and populates it with currently enabled themes/modules.
• each time a new theme/module is enabled/disabled, the system.extensions.json config file is updated to reflect the currently enabled extensions on the site.

(*) naming suggestions welcomed, but I don't think that calling it core.extensions like D8/D9 makes sense - there will be contrib modules/themes listed there as well.

I have not added the enabling functionality yet, since I don't know where it should go. I thought about these places, but looking for advise from people that know better:

• registry_rebuild() (is that a good place, or is it "expensive" to put things here)
• module_load_all() ...but then what about themes? It feels odd to add a theme enabling functionality in a function called module_*
• system_rebuild_module_data() / system_rebuild_theme_data() ? 🤷🏼

I don't feel that there would be a lot of value if we were enabling modules/themes only during config sync (it would be a manual task). I would like things to work soon as system.extensions.json gets updated, since that's how things work if you update the rest of the config files (causes the changes to take effect immediately).

[ghost]

an update hook creates a system.extensions.json config file*...

Don't we refer to modules/themes/layouts in Backdrop as 'projects'? Maybe enabled.projects.json, or enabled_projects.json... I think 'system' also seems to imply 'core' (at least to me).

[klonos]

Good suggestion @BWPanda 👍🏼 ...I went with system. because the update hook was in system.install, and the majority of the rest of the code changes were in files under /core/includes, which also falls under "system" AFAIK.

I agree that using the words "enabled" and "projects" makes sense, but I'll wait for more feedback from others before I start changing things.

[docwilmot]

Could someone explain under what circumstance this would be helpful? On first install? After youve installed and want to add more modules? I'm not getting it.

[stpaultim]

Could someone explain under what circumstance this would be helpful?

I think that this MIGHT be an offshoot of the "config recipes" discussion. It would be nice to package config files that enable new features on an existing site. If these new features require a specific module, this would allow the config recipe to turn on the module.

I'm not sure if @klonos has other things in mind.

[klonos]

@docwilmot there are a few benefits that I can think of:

• D8/D9 feature parity (from a marketing/selling point)
• allowing site deployments from config, which can be used in SaaS hosting platforms
• allowing locking of which modules are allowed to be enabled and which are "enforced" or disallowed, via config overrides in settings.php:
  • $config['system.extensions']['password_policy'] = 1; (prevents disabling security-related modules)
  • $config['system.extensions']['devel'] = 0; (prevents enabling modules on production when they are only required when developing locally)

[indigoxela]

TBH, I'm also having some trouble to understand the purpose of this feature. (D8/D9 parity is meaningless to me. If I want to use Drupal, I use Drupal.)

I think that this MIGHT be an offshoot of the "config recipes" discussion.

If that's the case, then I'm aboard, but I don't see (yet), in how far recipes rely on automagic enabling. I'd need "the big picture" explained.

What concerns me with the system.extensions.json approach is, that two different things do the same job (db table and config). I see a good chance, that we run into trouble when these two get out of sync and provide contradictory info.

@klonos many thanks for your concrete examples, but I actually doubt that these are 80% use-cases anyway. For deployments there's the installer. Locking the install of extensions via settings.php? Isn't that a completely different use-case? In how far is that related to "install/uninstall via config"?

[klonos]

D8/D9 parity is meaningless to me. If I want to use Drupal, I use Drupal.

Correct. Having said that though, during past outreach/marketing weekly discussions, it was mentioned that site owners that are considering their upgrade path from D7 are sometimes hesitant that Backdrop is lacking features/versatility compared to D8/D9. Also, it has often been mentioned that the argument comes up from time to time that "Backdrop is not meant for large-scale/enterprise projects". Having D8/D9 feature parity provides more confidence, and makes it easier for digital agencies to sell Backdrop. So it is important from a marketing point.

I see a good chance, that we run into trouble when these two get out of sync and provide contradictory info.

I was actually thinking of this as I was working on the PR. The status of modules is configuration, so I feel that we should eventually move it out of the database. Such a change would be too radical though, and I don't want this feature to "suffer" because of that, and become yet another thing that we push to 2.x

Locking the install of extensions via settings.php? Isn't that a completely different use-case? In how far is that related to "install/uninstall via config"?

Not a different use case, rather than a side-feature, that will come as a bonus out of this...

Sorry if I am not doing a good job describing the use case 😓 ...I'm sure that @BWPanda understands it now that he's working in the GovCMS platform, so perhaps he can chime in to help with explaining it. In the meantime, let me have another go at explaining what Drupal 8/9 allows in that context:

Customers in a GovCMS SaaS plan are not allowed direct access to things like the modules or the permissions pages via the admin UI (for security compliance reasons), and they do not have access to user1, nor SSH access to their servers/environments. They also cannot use custom or contrib modules that are not included in the GovCMS distribution - only theme files are allowed to be pushed (and config files if they are using GovCMS 8/9). This means that they cannot directly enable/disable modules.

In GovCMS 7, customers need to raise tickets with the Service Desk, to enable/disable modules for them. This causes:

• overhead/cost for the GovCMS team (which manages 400+ sites, with the list constantly growing)
• frustration to the customers and their developers, that demand more flexibility and the option to be able to self-serve for such tasks, instead of having to wait for the GovCMS Service Desk team to action their request (during business hours).

Those in GovCMS 8/9 though are able to self-serve: they can have their developers clone their site on their local, enable and configure any modules they need, and then export their configuration and push it to their GitLab repo. Webhooks are then triggered, which initiate deployment scripts that automatically do a config import, and enable any modules as configured in the core.extensions.yml config file that was pushed.

That is well and all, but there are modules that:

• should never be enabled in the SaaS plan, because they have a performance impact on the entire platform, which means increased cost (dblog for example should never be on, because syslog is used instead, to push logs to a centralized logging system)
• should never be enabled because they are unnecessarily consuming resources (again, cost), such as the update module (the GovCMS team manages updates for all SaaS sites)
• should never be enabled on production (things like devel)
• should always be enabled, for security compliance reasons (clamav, password_policy, tfa, etc.)

So there is need to allow customers some flexibility, while at the same time also allowing the platform owner to be able to lock certain things. Drupal 8/9 allows for that, and so could we. In fact, I actually think that we are in a better place, because our config is the source of truth ...excluding the case of modules/themes as is now, which is what I am trying to address in this issue here.

[findlabnet]

How many real users have requested this functionality?

[klonos]

One 😅 ...if you're asking how many people have asked or expect Backdrop to have feature parity with D8+ and be considered a good fit for larger/enterprise projects, then plenty I guess.

[findlabnet]

Does this mean that our target audience is large enterprises? Looks like I'm missing something.

[indigoxela]

Maybe one thing needs discussion elsewhere, or a least a clear position on Backdrop side: do people really want Backdrop to be like D8/D9? I doubt it. If I need the "big deal", I'd still use Drupal, but my projects are almost never in that range of customers.

Personally, I prefer Backdrop, because it is not like D9. It's fast, it's easy and quick to set up, yet flexible enough for more ambitious projects. (And I don't have to rebuild the whole project every four years. :wink:) IMO Backdrop shouldn't try to blindly adopt things as they're done in Drupal nowadays, because people considering Backdrop do it often, because they decided against Drupal.

@klonos I understand your point of view, but you're actually using Drupal a lot more than Backdrop and the projects you work in are clearly "big business" and "big budget", hence not what Backdrop tries to and should cover. The small agencies are our target. The kind of site builders, that were very happy with D7 but got driven away from Drupal because they don't consider it a good fit for their projects anymore.

[ghost]

I like and support the argument for not being like D8/9 just for the sake of it. There's an advantage in being different, even if that means Drupal has features/abilities Backdrop doesn't.

[klonos]

How many real users have requested this functionality?

Since when did that stop us from adding useful features to Backdrop? If that was an argument, then I can think of many other things that should never have gotten in. In fact, that would be an argument for Backdrop to not have ever been created in the first place.

Does this mean that our target audience is large enterprises?

It means that our target audience does not exclude enterprise, or larger projects. If features like this one is what's making D7 site owners and developers choose Drupal instead of Backdrop, then I would like that to not be a factor if we can help it.

Being like Drupal "just for the sake of it" is one thing, and having the same features as Drupal without the added complexity is another. We don't have to be a 100% match, but we don't have to not be like Drupal "just for the sake of it" either.

Is everyone really against this feature? If that's the case, then I'm happy to stop working on it and close the PR. If we change our mind in the future, at least we won't start from scratch. If people want to move this to contrib, then that's fine too (I won't, because I have no use for it there).

The thing is that the more I work with Backdrop, the more I see that there are some inconsistencies that simply do not make sense to me. We have introduced CMI as a big feature (it really is), and that means that ideally content and caches should go in the database, and things that are configuration should go into config files. The module/theme state is configuration, yet it's oddly still something that we have kept in the database 🤷🏼 ...but since this is not what I'm proposing here, lets park that thought.

For those of us working with a config workflow in place, do you really want to need to push config changes, and then also having to manually enable modules (2 tasks), instead of just pushing config and be done with it (1 task)? What is the argument against simplifying that use case? Would having that feature in place (which is what I'm proposing here) be complicating the work of "simpler" sites somehow? If it is adding an obvious, considerable performance hit, or if the code complexity increased to a point where it warranted the feature not worth it, then I'd understand.

[laryn]

I think I’m with Klonos here. I remember a few years back when a large university was considering Backdrop for their hundreds of smaller Drupal 7 sites the mantra was that we prioritize the 80% but individuals or orgs were free to submit improvements that would benefit enterprise users — so long as they didn’t have a detrimental effect for the 80%. It seems that is the case here, doesn’t it? The average user doesn’t need to know about this and it doesn’t affect them, fine! But some users may find it very helpful.

Unless there’s an argument that this increases code complexity to a large degree or affects performance negatively… it seems very reasonable to me.

[stpaultim]

I seem to recall having run into situations on several occasions when this would have been very helpful during deployments and I have heard from others that they ran into the same problems.

I'm not 100% certain that this would resolve those issues, but I'll try to do some testing soon to figure that out. I'm fuzzy on the details, because I've been working around this issue for the last 1-2 years.

I also think that this will open up some possibilities for config recipes, but again I'm not in a position right now to detail exactly how this would help (so I could be wrong). I'll think about it some more.

[docwilmot]

I was more asking about the practicality rather than benefits per se. I think your response:

allowing site deployments from config, which can be used in SaaS hosting platforms

seems the main reason after all. It sounds like this would work like an install profile's list of dependencies[], and as such it isn't really an unprecedented idea IMO. This really would be for new installs only right? I am not seeing a practical application for sites that are already up and running. I dont think if your site is live and you manually deleted something from that list that it would/should disable the project automatically? Am I right there?

A few questions though:

• would the installer fetch the modules on the list, or would you still have to upload/download them?
• would we deprecate the dependencies[] list in profile info files?
• similarly, would a module or a "recipe module" be able to have such a list in its config too? Or would this belong solely to core?

[klonos]

...I think your response:

allowing site deployments from config, which can be used in SaaS hosting platforms

seems the main reason after all.

Yes, that ^^ ...but also this:

For those of us working with a config workflow in place, do you really want to need to push config changes, and then also having to manually enable modules (2 tasks), instead of just pushing config and be done with it (1 task)?

[klonos]

• It sounds like this would work like an install profile's list of dependencies[] ...

  Not exactly. Updating the dependencies[] list in an installation profile "on the fly" (after the site has been installed) does not trigger enabling/disabling of the additional/removed dependencies. You can test that:

  • install a vanilla Backdrop instance
  • edit /core/profiles/standard/standard.info, and add dependencies[] = contact and save
  • check the modules list on the site -> the Contact module is not enabled 👎🏼

  ...whereas what I'm trying to achieve here is:

  • install a vanilla Backdrop instance
  • on your local enable the Contact module, either:
    • via the admin UI, or
    • via drush, or
    • by directly editing the system.extensions.json file
  • push/copy your config files to the server
  • check the modules list on the site -> the Contact module is enabled 👍🏼

• would the installer fetch the modules on the list, or would you still have to upload/download them?

  I was thinking about checking system.extensions.json for any entry that is not already enabled, and then rely on native/existing functions, like module_enable(). What happens now if you call module_enable() passing a module that does not exist in the codebase? Does Backdrop fetch it and then enable it? Does it skip it? ...anyway, I imagine that it would work the same as that use case; I haven't reached that point yet though. All the code in the PR currently does is to watch for any enabling/disabling of modules or themes, and then updates the system.extensions.json config file accordingly - nothing more. Next step would be to implement the enabling of modules/themes, and I was thinking to consider disabling/uninstalling as well next (if it turns out to not be complicated/risky etc. - if it is, then either separate follow-up issue, or not allow that functionality at all).

• would we deprecate the dependencies[] list in profile info files?

  Nope, not deprecating that - this feature is not a replacement for dependencies[]. I think that dependencies[] allows for different scenarios and use cases. Not the same as what I'm trying to achieve here.

• similarly, would a module or a "recipe module" be able to have such a list in its config too? Or would this belong solely to core?

  Nope (unless we want that - I haven't thought about it 🤔). Recipe modules would still use dependencies[] in order to enable any modules they require. But you could use this feature here to trigger the enabling of the recipe module by pushing a config change to the site. Then module_enable() would take care of enabling anything specified via dependencies[] in the .info file of the recipe module.

[docwilmot]

push/copy your config files to the server check the modules list on the site -> the Contact module is enabled 👍🏼

I could appreciate this as a list of modules to check and enable to install, but not to disable. The idea of something as potentially destructive as disabling (a bunch of) modules in the background does not sound like a good idea.

What would trigger the sync? Cache flush? Cron?

We'd still need to consider profile dependencies carefully; if you had a config list with a totally different list of projects from the list of dependencies in your profile, what would/should happen?

[klonos]

I could appreciate this as a list of modules to check and enable to install, but not to disable.

Agreed 👍🏼 ...lets examine that possibility separately, in a follow-up.

What would trigger the sync? Cache flush? Cron?

Ideally, I would like something less manual, and more immediate than having to wait for a cron run. If something during bootstrap is too expensive, then sure, I could settle for that.

if you had a config list with a totally different list of projects from the list of dependencies in your profile, what would/should happen?

Nothing different than the current behavior, besides enabling modules in system.extensions.json that are not already enabled.

dependencies[] defined in the .info file of installation profiles are not "enforced" after the installation stage. You can test that like so:

• install a vanilla instance of Backdrop (the Standard installation profile)
• disable and uninstall any of the modules in the dependencies[] defined in https://github.com/backdrop/backdrop/blob/1.x/core/profiles/standard/standard.info (something like Views UI, or Update, or the Dashboard)

[stpaultim]

In case there are still doubts about the use case for this PR, I did some experimenting and had a conversation with @klonos. Here is the common use case that I think many basic users would encounter. Deploying a site using Backup and Migrate.

https://docs.backdropcms.org/documentation/deploying-backup-migrate

We have instructions on our docs site on how to deploy your site using Backup and Migrate. Those instructions suggest that the user:

1. Deploy the code
2. Go through the install process
3. Enable all the modules on your destination site that are enabled on the source site, before importing config.
4. Then after you an imported config, you should import your database with Backup and Migrate.

While this process is not something I would normally use and I expect most developers would not use this in general, I think it is something that lots of beginners or site owners are likely to try and even makes sense in some fringe cases for someone like myself.

I tried this process with two of my sites and found that the process of enabling modules is quite tedious and prone to mistakes. @klonos points out that this PR would remove that step in the process.

I hope that this is helpful in providing at least one relatively common use case.

[stpaultim]

I could appreciate this as a list of modules to check and enable to install, but not to disable.

Agreed 👍🏼 ...lets examine that possibility separately, in a follow-up.

Given that the prospect of disabling modules is most controversial and there seems to be agreement to treat that later, should we change the title of this issue?

Also, I assume that this is not ready for testing yet (as there is no needs testing tag).

[stpaultim]

Discussed during meeting today: https://youtu.be/CoXHmSUGsXg?t=925

[klonos]

Perhaps adding _module in all config files (#1811) would make this issue here moot.

[jenlampton]

Perhaps adding _module in all config files (#1811) would make this issue here moot.

I think it would still be nice to be able to enable/install modules on sync, but we may not end up needing the separate config file.

[yorkshire-pudding]

I would like to add my support for this. I'm trying to script deployments of development site and running into issues with config import. I would like to be able to specify to enable modules that my git has put the code in for.

I'm currently thinking about how to export a list of config and then use that to drive bee en .... but a dedicated function to work from a file would be even better.

[laryn]

I was just thinking about this issue again during a deployment that required config syncing and failed because there was config in staging for modules that were not enabled on production. It seems reasonable to me to assume that if you're syncing config that requires a certain module, and that module's code exists in the modules directory, automatically enabling the module (and adding a message to the user that it has been enabled) is expected and reasonable behavior.

[eigentor]

Coming from Drupal I can second this is not only useful, but I would say expected. When importing config I again and again have to stop and enable some modules first. So are there any patches or concepts one could improve upon? Am ready to give it a shot if there is some kind of consensus if it should be a separate config file or not.