Config file to expose all environment variables declaratively

[ChipWolf]

Enhancement Type

A completely new feature

Describe the enhancement

The number of configuration options and environment variables is now quite large and unwieldy. For more intricate setups, it'd be valuable to consolidate the dozens of environment variables down into a config file.

• I imagine the environment variables would remain for backwards compatibility and potentially be deprecated until the next major version.
• All logic that depends on environment variables could transition to reading the config file.
• The logic behind the config file could contain inline documentation that could be extracted by CI into user-facing documentation (#1440)
• A helper function would bridge the gap between new and legacy, containing a mapping of environment variables to YAML paths.
  • The helper could generate/update the new config if legacy environment variables are exposed.
  • The helper could possibly inform the user the config file has been generated/updated.
  • The log could also detail which legacy environment variables were converted.
  • The log could also state the legacy variables can be removed from the user's container spec if they wish.

[itzg]

The number of configuration options and environment variables is now quite large and unwieldy.

I think it's time to start consolidating everything down into a config file.

I appreciate the concern, but that is a huge undertaking and would be immensely disruptive to all of the existing users.

In any case compose files, deployment manifests, and helm values are already "config files". What advantages do you propose that a formal config file format would provide?

Additionally, the documentation for this container is likely worth moving to Mkdocs or similar to make it more navigable and accessible.

Please open a separate issue for this.

[ChipWolf]

A config file could be used instead of the environment variables, it could be an option rather than a requirement.

.itzg-mc

minecraft:
  eula: true
  version: 1.16.4
  configuration:
    motd: "My server"
    whitelist: true
airplane:
  version: PURPUR
  flare: true
jmx:
  enabled: true
  host: myhost.com
autopause:
  enabled: true
  timeout:
    init: 600
    knock: 120
  period: 10
  interface: eth0
misc:
  gui: false

...

[ChipWolf]

@itzg does this issue need to be moved to the helper repo?

[itzg]

This is still the right place for it.

[ChipWolf]

In any case compose files, deployment manifests, and helm values are already "config files". What advantages do you propose that a formal config file format would provide?

The primary benefit I see is in idempotency. For example; if it were viable to ship the proposed .itzg-mc config as part of generic pack builds, the variables could be iterated on and versioned declaratively.

Modpacks generally have a hard requirement on specific configuration - a file-based config could eliminate drift between the desired and actual state of the container from the pack's perspective.

For example, if a mod update requires a new version of Forge; the generic pack change to update the mod could also bump the Forge version.

[itzg]

The overall feature is starting to seem more reasonable.

However

Modpacks generally have a hard requirement on specific configuration - a file-based config could eliminate drift between the desired and actual state of the container from the pack's perspective.

This is already supported by some of the modpacks (not as generic packs) and I'm planning on broadening the modpack driven setup support.

So let's focus this particular issue/enhancement on non-server-type/version configuration.

[brandon1024]

@ChipWolf Wouldn't the --env-file option of docker-run provide what you're looking for? Rather than configuring each environment variable through -e <var> options, you could maintain your own environment config file.

--env-file=file
       Read in a line delimited file of environment variables. See Environment note below for precedence.

It wouldn't hurt to keep a template file directly in the project with all supported variables and short descriptions for each variable.

[ChipWolf]

@ChipWolf Wouldn't the --env-file option of docker-run provide what you're looking for? Rather than configuring each environment variable through -e <var> options, you could maintain your own environment config file.

--env-file=file

       Read in a line delimited file of environment variables. See Environment note below for precedence.

It wouldn't hurt to keep a template file directly in the project with all supported variables and short descriptions for each variable.

Not entirely, I'm proposing a config file for the container, Docker is not the only container engine

[brandon1024]

What other OCI implementations are out there? :sweat_smile: I use podman but this flag is supported by every implementation I'm aware of (docker, podman and docker/podman-compose, and even k8s with config maps or secrets).

If you'd prefer not to use --env-file, another solution would be to have the entrypoint source an env-var file at startup.

-e SOURCE_ENV_FILE=true
-e ENV_FILE=/<some-mounted-file>

[itzg]

I think there's a lot of merit in @brandon1024 's suggestion. Certainly the command line syntax shown is specific to docker run, but every container platform I have worked with supports "environment variable files". That ubiquity isn't surprising given industry guidance like https://12factor.net/config .

Even without the container engine supporting it, the startup script could load an environment file like

set -a
source "${someEnvFile}"
set +a

I'm still not sure what a proprietary config file format would gain. A user starts with a blank file and then what? They need to refer to documentation like today.

Are you envisioning a file that the user grabs from a baseline copy that is pre populated with all possible variables and the user adjusts the ones that apply to their scenario?

Or is it yaml based in order to bring structure and organization to the configuration rather than a flat set of environment variables?

[itzg]

Oops, you beat me to the same point @brandon1024 😀 ...I kept getting distracted while writing out my comment.

[brandon1024]

@itzg haha no worries mate!

If it were up to me, I would steer clear of a yaml config file :sweat_smile: To me, I see so much unnecessary complexity with very little gain in what @ChipWolf proposed. Yaml needs to be parsed meaning we would need to package a parser of some kind, and I wouldn't envy anyone that needs to write the (non-trivial) scripts to convert between legacy variables and yaml.

And to to reformulate what @itzg hinted at, an env-var file is self-documenting; a yaml file would not be. I will agree that the configuration options of this container is a bit overwhelming at first glance, but I think that could be better solved with improved structured documentation. I digress :)

Are you envisioning a file that the user grabs from a baseline copy that is pre populated with all possible variables and the user adjusts the ones that apply to their scenario?

I actually really like this idea; we could provide a baseline template env-var file at the project root similar to a handful of projects out there (dnsmasq, kafka, redis, sshd, etc).

[ChipWolf]

The major benefit I can see from using something like a yaml config is due to the feature hierarchy.

Variables are relatively linear, but given the number of variables defined for each feature the structure of a hierarchical config is appealing.

FOO: true
FOO_BAR: true
FOO_BAR_BAZ: banana

foo:
  enabled: true
  bar:
    enabled: true
    baz: banana

As long as there's consistent naming scheme for variables, a transformer between the two would be universal. See how LuckPerms handles this https://luckperms.net/wiki/Configuration#environment-variables

Given the existing helper binaries in the image, there shouldn't be any concern about additional dependencies to achieve something like this.

---

In terms of config, yes I agree application config should be stored in the environment per 12factor, but applications generally ship with defaults and in the case of generic packs - they're effectively a template.

Imagine you have a modpack distributed for use with the itzg image & you'd like to ship it with some defaults. As proposed here, the pack itself could contain those defaults, then the consumer of the pack could choose to override them with variables if they wished.

Right now, a trivial change to a pack can easily become a breaking change to the consumer (i.e. your server config):

For example; you interpolate a config variable to be used with REPLACE_ENV in your pack. The addition of a single variable to the pack is now a breaking change/major version bump as the consumer's environment config must be changed to support the new variable or it will ~~be replaced with null during initialisation~~ fail to start.

If a default config was included in the pack containing the default value for the new interpolated option; the change to the pack could be a patch/minor version bump instead & wouldn't require a consumer of the pack to change anything.

I understand the perspective where this might not seem necessary, especially if there's only one person in your environment handling config/packs/servers; but consider more complex situations:

• variants of a server may be using the same pack and each have to maintain a full duplicate of the environment variables (risk of drift/unpredictability)
• many people running a network could have focus on different areas (a change to the pack by someone - such as one described above - may, for example, add overhead/delay for another person who normally maintains the servers)
• groups sharing resources who need to manually keep sync

You wouldn't want your users to edit their own config for each minor/patch version update to your application, that's effectively what I hope this RFC can find a solution to.

[itzg]

The image already enforces defaults for all variables. Yes, I agree a yaml file is cosmetically more readable/manageable but an env file is just as portable for what you're describing.

To clarify something: I will not be modifying all of the image scripts and software to start reading from config files. They will always leverage environment variables (or pass thru command line args). So, if there were such a yaml mechanism it would only be a frontend to map into setting environment variables.

Anyone is welcome to develop and maintain such a mapping tool, but I will not be taking that burden into my repos. The Helm charts have already worn out my patience.

[brandon1024]

@ChipWolf if you really want to configure the server with yaml-based configuration, maybe consider creating a project that uses this project as a base layer and provides the functionality you're looking for :-)

[ChipWolf]

@ChipWolf if you really want to configure the server with yaml-based configuration, maybe consider creating a project that uses this project as a base layer and provides the functionality you're looking for :-)

Happy continuing contributing here, itzg has admirably poured enormous amounts of effort into a phenomenal resource & it's very well maintained; a fork would be crazy overhead.

I'd be happy to PR anything the community lands on but I wasn't wanting to spend hours on a huge refactor before an RFC.

I imagine as more of the complexity was being dissolved out of the shell scripts into the mc-image-helper, the vision is to continue in the direction of [mostly] eliminating the scripts at some point?

Happy with an env-file approach [even as a stopgap] if we're not wanting to lift out any significant chunks of logic into the wrapper/helper anytime soon.

[brandon1024]

a fork would be crazy overhead.

Apologies, maybe my wording wasn't super great. I didn't mean to suggest a fork of the project (agreed, forking would be a nightmare), but rather just a docker image that is based on itzg/minecraft-server that includes your yaml configuration capabilities:

FROM itzg/minecraft-server

COPY <your configuration scripts>

ENTRYPOINT [ "<your entrypoint>" ]

[itzg]

I imagine as more of the complexity was being dissolved out of the shell scripts into the mc-image-helper, the vision is to continue in the direction of [mostly] eliminating the scripts at some point?

Yes. In fact, I could backpedal a little bit and point out that support yaml driven configs in mc-image-helper is actually quite doable, but still an additional maintenance vector I wouldn't want to maintain myself.

FWIW, @brandon1024 I knew what you meant about leveraging the base image 😀 and agree that is the kind of "frontend" that I could see enabling a conversion to environment variables.

[ChipWolf]

an additional maintenance vector

I'd be happy to maintain something like this, but with the proposed design here, it would become the only vector for maintaining variables - there wouldn't be any additional ongoing overhead so to speak.

There'd of course be a one-time effort to implement a config handler into mc-image-helper and write a transformer to map the "legacy" variables to the new format; but after that point, any new variables would be exposed automatically per the LuckPerms example above and could still be consumed by shell scripts.

To alleviate any concerns of additional overhead, I envision the shell scripts would reference the environment variables directly as they do now, with a view of mc-image-helper able to absorb them as and when required.

Within the design I'm proposing; users would be able to configure the service with environment variables or via the config file (perhaps even both where one overrides the other), variables would be normalised and populated in the environment per the LuckPerms example no matter how they are set thanks to the transformer, and the scripts wouldn't need a refactor to support this. It wouldn't be a breaking change, it'd just set up the service to support more flexibility and a better future state.

The config schema would be versioned, the schema version would be referenced in the config itself, and the transformer could be updated to ensure backwards compatibility is maintained. If any breaking changes to the schema were made in future, users would not need to update their existing configs as the config handler in mc-image-helper would see the config version, transform it to the new schema internally, and optionally throw a soft warning to the user that they should update given a good reason to do so.

[itzg]

I appreciate the offer, but I really don't want any part of that in any of the repos I maintain. These are my projects and I have particular design strategies throughout -- this basically changes everything. I'm sorry if I misled you by entertaining the initial discussion on this topic.

You're welcome to implement all that in adjacent tooling, forked repos, extending the image, etc.

I'm still living with the mistake of owning the helm charts (reminded by the latest issue report there) and I am already spending too much of my personal time on all of these things with very little compensation.

[itzg]

I will offer this concession:

I have been needing to maintain (and will build a tool to generate) a manifest of all environment variables used by the image, their defaults, category, and brief description when feasible. I'll either produce a json or yaml file that will be committed and baked into the image. I might add an mc-image-helper command to dump the file in various formats: json, yaml, CSV, markdown table, etc.