Persist configurations set via console

[flyingsilverfin]

Problem to Solve

To build up on #6460, we wish to enable users to persist configurations to the database. This is especially useful in TypeDB Cluster, where there is then one place to configure all servers at the same time. To do this, we will need to write the configurations the user set into a replicated database (a regular database in TypeDB).

Proposed Solution

We extend the solution in #6460 to include persistence, where configurations are persisted into a database under the hood.

User Experience

The user will be able to set persisted configurations via console. This will require an extension to protocol design, and for example require authentication on cluster.

./typedb console
> configure

This should open an editor containing the current configurations, as a YAML structure. The user can edit this file and save, which submits it to the server. The server should persist any changes.

Settings that have not been persisted, should continue read from non-persisted defaults. This allows us to release new distributions and update configurations the user hasn't specifically set.

The YAML file the user sees should make clear which parameters are reading from the default values and will not be persisted unless changed.

In Core, the changed settings should persist to the internal config database, and in cluster it should do the same, which the current master of the database will handle behind the scenes, replicating any changes.

The user must reboot all the servers for the changes to take effect. Hot reloads can be a feature in the future (eg. we could annotate configurations with what parts of the server need to be reloaded/reset and only reset the subsystems affected by a new parameter.)

Persistence

To enable persistence, we need to write the data somewhere. The proposal is to restructure the storage layer into the following:

/server
  |- /config  (contains persisted configurations)
  |- /data     (contains all user data, including user management, etc.)
     |- /databases  (all user databases)
     |- /users  (or something else that encompasses cluster management data)

We separate /config and /data so that the user can persist into the configuration where the data is to be stored (for example, on a different mounted volume).

The user should also have the option of storing the /config directory outside the server distribution, with a command line flag only, which is useful if in a read-only container of some kind:

./typedb server --persist-config=/path/to/user/dir

This will have to be provided each time, or the server will not have access to the user's configurations! To prevent silly mistakes, the user should get a message at bootup saying the server is using only defaults if there are no persisted configs found.

Requirements

To implement this effectively, cluster should already have implemented cluster membership change - only one machine can set up the config database the first time. See issue TODO

Outcome

After this is implemented, we should be able to:

• retain the behaviour of setting a config for the duration of the server bootup via command line (on cluster, a single node's command line args will override the cluster-wide settings, useful for making a single server persist to a different directory, for example)
• use console to set persisted configurations, in a user-friendly way
• in cluster, setting cluster-wide settings this way should propagate to all the machines
• we don't implement hot-config reloads, but require server reboot on changing persisted config

[lolski]

Requires persistent, replicated config to be implemented in cluster: https://github.com/vaticle/typedb-cluster/issues/375