Clean up configuration and command line handling

[msimberg]

The runtime configuration is currently in a pretty messy state with many ad-hoc workarounds. It parses configurations multiple times, environment variables provide defaults instead of overriding configuration options, and many options are ignored if put in the wrong place (see #743). I would propose that we streamline the handling of configuration and command line options.

The following is an attempt to summarize an "ideal" flow of processing options, attempting to be as close as possible to the current behaviour. A later step always overwrites options set from previous steps.

• Read a default configuration hardcoded in a source file or in a separate configuration (read-only) file stored on disk
• Read a configuration file from PIKA_CONFIG if set, otherwise read a configuration file from ~/.config/pika/config.ini if exists
• Read configuration options from PIKA_INI if set
• Read a configuration file from --pika:config if set
• Read configuration options from --pika:ini if set
• Read configuration options from other environment variables, such as PIKA_PROCESS_MASK
• Read configuration options from the command line, such as --pika:process-mask
• Normalize, expand, replace options if that's something we want to support (we currently support referring to other configuration options)

An alternative workflow collapses some of the handling of configuration files:

• Read a default configuration hardcoded in a source file or in a separate configuration (read-only) file stored on disk
• Read a configuration file from --pika:config if set, otherwise from PIKA_CONFIG if set, otherwise from ~/.config/pika/config.ini if exists
• Read configuration options from --pika:ini if set, otherwise from PIKA_INI if set
• Read configuration options from other environment variables, such as PIKA_PROCESS_MASK
• Read configuration options from the command line, such as --pika:process-mask
• Normalize, expand, replace options if that's something we want to support (we currently support referring to other configuration options)

A third, yet simpler worflow that leaves out the --pika:ini/PIKA_INI options could look like this:

• Read a default configuration hardcoded in a source file or in a separate configuration (read-only) file stored on disk
• Read a configuration file from --pika:config if set, otherwise from PIKA_CONFIG if set, otherwise from ~/.config/pika/config.ini if exists
• Read configuration options from other environment variables, such as PIKA_PROCESS_MASK
• Read configuration options from the command line, such as --pika:process-mask
• Normalize, expand, replace options if that's something we want to support (we currently support referring to other configuration options)

I like removing the duplication between PIKA_INI and other environment variables (likewise --pika:ini and other command line arguments) because it's not clear to me why one should have precedence over the other. The same information can be passed to pika with both mechanisms, assuming that every option has an environment variable and command line option.

Further, we could also remove the configuration files as well and further simplify configuration. The workflow would then look like this:

• Read a default configuration hardcoded in a source file or in a separate configuration (read-only) file stored on disk, but in an unspecified format
• Read configuration options from other environment variables, such as PIKA_PROCESS_MASK
• Read configuration options from the command line, such as --pika:process-mask
• Normalize, expand, replace options if that's something we want to support (we currently support referring to other configuration options)

Finally, if we keep the configuration files I would try to use toml, yaml, or json so that we can use a third-party library for the parsing. I personally like toml and it's the most similar to the format we currently use, but I'm not set on toml.

Ideally, we should in the end be able to define a configuration option as a tuple:

{"pika.name_of_configuration_option", "default_value", "PIKA_NAME_OF_ENVIRONMENT_VARIABLE", "--pika:name-of-command-line-option", command-line-option-type, "description"}

Preferences or thoughts?