rpk: add ability to morph clusters from dev to prod

[emaxerrno]

given the existence of rpk redpanda start --sandbox-style family of commands, it would be great to have an rpk command - adatabase of profiles of a production workload.

Effectively once we introduced centralized configuration, I think morphing clusters from dev to prod broke.

I propose a style of

rpk cluster config set --well-known=production

to override the old settings like partition count, core count, fallocation size, disk reservation, etc, etc, at runtime on a cluster.

What's nice we can even teach it multiple profiles like

rpk cluster config set --well-known=redpanda.cloud

These are just profiles, so there is no secret sauce there. and the settings are really helpful as a bundle

[twmb]

Is the expectation that people will also fixup their deploy scripts to remove any --sandbox or other dev specific flags they have already set?

[emaxerrno]

regardless of where they are, think of it like git reset --hard for settings.

[jcsp]

I'm not sure this makes sense (I don't like the idea of transforming sandbox into prod system, the general idea of configuration profiles is fine but doesn't belong compiled in to rpk)

Let's step back one. If on production ready environment, why would you ever have set system up in sandbox mode? I think the only reason is either A) mistake or B) because rpk defaults to developer_mode.

Case A we should fix at install/setup time: whether that's rpk or something else, it should validate the environment, and if it meets requirements for prod then run the system with the prod config (which is just redpanda defaults, remember).

Case B goes away when the system is being set up with operator or ansible, which write their own configs and don't default to developer_mode=true. It can also go away for rpk cluster config init if we choose to change it.

Also, remember that rpk redpanda start is going away from production deployments at some point (https://github.com/redpanda-data/redpanda/issues/4794). Once that happens, someone using rpk redpanda start will signal they are not in a production setup, so it doesn't make sense to talk about how we morph a system like that into prod mode.

Also, this whole thing undermines the concept of a sandbox cluster being ephemeral/disposable. Switching off fsync is okay for an ephemeral cluster. Switching off fsync on a cluster that you intend to put into production later is waaaay not okay. If --sandbox actually means "This cluster is not prod yet" then that's a profound change to the definition of the command.

All that said, if we did want a way to un-sandbox-ize a redpanda cluster (which I am not convinced we do), then it should not be an rpk-known set of config values, it should be for rpk to simply clear the properties that it set in --sandbox, so that Redpanda's defaults (chosen for production) come back into effect.

[dotnwat]

if we did want a way to un-sandbox-ize a redpanda cluster

maybe we could provide an rpk export / rpk import command for the rare case that someone wants to un-sandbox-ize. export data, wipe cluster and make production ready, then import.

[jcsp]

maybe we could provide an rpk export / rpk import command for the rare case that someone wants to un-sandbox-ize. export data, wipe cluster and make production ready, then import.

Yeah, this kind of thing is a much more robust approach than modifying clusters in place from a "dirty" starting point. Near term it's the kind of thing where we might just direct people to use mirrormaker to copy their data.

[twmb]

Closing in favor of the export/import idea, which is a follow up project for Tiered Storage