[feat]: Allow user to see new config options without going into Needs Config

[chrisguida]

Currently there are only two options for updates:

1. Automatically give the user default settings, don't stop the service, and don't tell the user about anything that has changed
2. Delete any invalid keys from the config, stop the service, put the service in needs config, show the user the new keys, and give them the default settings when they hit save.

There is no way to show the user what has changed in the config without grinding their service to a halt and requiring their intervention. This is very suboptimal for uptime-sensitive services such as lightning nodes. The user does not want their node to turn off and remain off until they intervene. At the same time, the user does want to know about important changes in the config.

There are several possible solutions to this. Either the backend needs to be aware of which things changed in the config, and store this somewhere until the user has viewed the changes, or the migration needs to return a list of which things changed so the frontend can display it at least once. I think I prefer the first option; perhaps this could be achieved with the help of a new field in the migration result type.

[MattDHill]

The current architecture is pretty flexible.

1. If there are any new additions to config, the service will go into "Needs Config", and new option will be highlighted as "New" and prefilled with defaults. This seems ideal to me. If you want to add new options, set default, and not require the user to view config, then maybe the release notes will do.

2. If there are no changes or only removal of options, the user updates with no interruption. This also seems ideal.

3. If a migration of data has taken place but not require the user to take action (or even care about what was migrated), maybe we can just include those changes in the release notes.

[chrisguida]

If there are any new additions to config, the service will go into "Needs Config", and new option will be highlighted as "New" and prefilled with defaults. This seems ideal to me. If you want to add new options, set default, and not require the user to view config, then maybe the release notes will do.

This is not ideal, especially for lightning nodes, because then the service stops until the user intervenes.

If there are no changes or only removal of options, the user updates with no interruption. This also seems ideal.

This is not ideal for important config changes that the user needs to be aware of.

If a migration of data has taken place but not require the user to take action (or even care about what was migrated), maybe we can just include those changes in the release notes.

This seems like a suboptimal solution given that we already have the infrastructure built out to highlight important changes in the config, making it much easier for the user to find where they potentially need to intervene. Release notes are not nearly as explicit, and cannot be made so.

[k0gen]

I would also prefer to just hit that UPDATE and expect the system to do auto configuration and starting up service for me automatically, especially where there is no or just minor config changes that for most users will be the suggested default anyways. If there were no config changes that should be the preferable solution. In case the update needs to change the config by adding options the service should be flagged and default setting should be applied to those option so the service can be started automatically. The flag would disappear the moment user enters the config and review new options. We should decide at what kind of scenario (in config or service change) the user intervention would be needed. But in this case this should be displayed as a warning before even starting right after clicking the UPDATE button.

[MattDHill]

Right now, package devs have two options.

(1) If they don't want the service to go into needs config after update, they can create a migration that satisfies the new config spec and include the list of changes in release notes.

(2) If the changes are important or require user selection, they can put the service into "Needs Config", which will prompt the user to set config and show them the new options, with or without prefilled defaults.

In addition to the above, I believe what @chrisguida is asking for is the ability to do number 1 but with a better display of the changelog than release notes. I believe what @k0gen is asking for is the ability to start a stopped service after update without needing to configure it but with a warning.

Assuming I understand correctly, I can say confidently that neither of these features would be easy to implement and might even require entire overhauls of the config system. I believe the flexibility already afforded the package dev is adequate for now, given the complexity of implementing these improvements. Happy to explain why this would be so difficult on a call

[chrisguida]

(1) If they don't want the service to go into needs config after update, they can create a migration that satisfies the new config spec and include the list of changes in release notes.

Right, but we already have the ability to highlight new config changes in the frontend, so we may as well put that to use. As I mentioned previously, release notes are not nearly as explicit. Highlighting the config changes inside the config is a great UX. Hiding config changes in the release notes feels like a stopgap solution.

(2) If the changes are important or require user selection, they can put the service into "Needs Config", which will prompt the user to set config and show them the new options, with or without prefilled defaults.

Right, but then the service will also stop. We need a way to show the user the config changes without stopping the service.

In addition to the above, I believe what @chrisguida is asking for is the ability to do number 1 but with a better display of the changelog than release notes.

I'm asking for number 1 but with the green highlighting that we already have on the frontend. Or, number 2 but without stopping the service.

As @k0gen said:

The flag would disappear the moment user enters the config and review new options.

This is exactly what I'm asking for. Highlight the config options only the first time they enter the config, or maybe give them a "dismiss" button so they can choose when to dismiss the new config option notification.

Assuming I understand correctly, I can say confidently that neither of these features would be easy to implement and might even require entire overhauls of the config system. I believe the flexibility already afforded the package dev is adequate for now, given the complexity of implementing these improvements.

The purpose of this ticket is not to be easy to implement, or fast. The point is to come up with an elegant solution to the problem. I don't mind if it takes a while, but I wanted to start the conversation.

But also, I don't think it will be a huge lift. Again, I think this could be implemented either by diffing the configs in the OS, or by having the migration return additional fields. Then the FE could just use its current green highlighting feature to relay this important info to the user.