Accessing environment variables

[yannham]

Is your feature request related to a problem? Please describe.

A feature request that came up several times on various channels is to be able to access environment variables. This is possible today by performing some pre-processing step (for example, a script dumping the environment to a file or converting it to a .ncl file that can be imported), or by using the customize command line, for example:

# file: config.ncl
{
  ENV_PATH | String,
  config = {
    new_path = "%{ENV_PATH}:/nix/store/x29idoa934zdoijgoin9nickel-1.8.1/bin",
  }
}

$ nickel export config.ncl --field config -- ENV_PATH=\""$PATH"\"
{
  "new_path": "/home/johndoe/.local/bin:/nix/store/x29idoa934zdoijgoin9nickel-1.8.1/bin"
}

The pre-processing approach has the obvious drawback of requiring everyone to come up with their own scheme and scripts for a task that seems to be common to many use-cases. The customize-mode approach can quickly lead to very long and unreadable command line (needing some form of scripting anyway, probably). In both cases, it's also not trivial to handles cases where the environment variable has special characters that might be interpreted in the Nickel string, such as ", escape sequences or %{. It can be solved practically by using a multi-line strings with a complicated enough delimiters, but then new lines and indentation can become an issue as well...

Having a standard built-in solution to this problem is reasonable.

Describe the solution you'd like

Using variables from the environment have the potential of breaking Nickel's purity if done wrong. If the result of the program can suddenly depend on unspecified environment variables in an uncontrolled way, it can become very challenging to ensure reproducibility of evaluation across different environments. Thus, whatever we end up with, we must ensure that which variables are taken from the environment (and from which environment variable) can be easily and immediately decided, without having to evaluate the program.

This precludes a simple approach such as a free-form dynamic std.env.get that can retrieve any value dynamically from the environment. An explicit mapping of some top-level variables to environment variables might be a better idea.

1. One simple solution would be to propose a shortcut for the customize-mode approach, with proper escaping, such as: nickel eval config.ncl --field config --take-from-env ENV_PATH PATH. We could also decide that the name of the Nickel variable and the environment variable are deducible one from each other, to end up with something like nickel eval config.ncl --take-from-env PATH CLASSPATH, which would be equivalent to nickel eval config.ncl -- ENV_PATH="escape("$PATH")" ENV_CLASSPATH="escape("$path")" (where escape is a bash functions performing the required escaping)

2. Another approach is to use a special file instead. Either specified on the command line, or fixed by Nickel, we would have say an env_var.ncl file with a mapping of the form:

   {
       my_path = "PATH",
       my_classpath = "CLASSPATH",
   }
   

   Either the mere presence of this file, or a command-line argument would cause Nickel to evaluate it, build the corresponding record {my_path = <value of PATH>, my_classpath = <value of CLASSPATH>}, and merge it with the original config.

There might be other solutions in between.

[suimong]

A few thoughts of my own:

1. The first approach is nice for ad-hoc usage / simpler cases, while the second one is more appropriate for programmatic usage.
2. The schema of env_var.ncl should not be too restrictive e.g. {_: String}, as there's the documented pattern of putting input variables under the inputs key, therefore:

{
    inputs.my_path = "PATH",
    inputs.my_classpath = "CLASSPATH",
}

should evaluate to {inputs.my_path = <value of PATH>, inputs.my_classpath = <value of CLASSPATH>}

3. there should be an escape hatch, e.g. an environment variable (NICKEL_ENV_MAP_FILE or similar) that allows user to specify a different name for env_var.ncl. The reason is that IMHO env_var.ncl is a name that is "not too uncommon" and susceptible of multiple valid interpretations. It just so happens that I use a similarly named file (env_vars.ncl) to store configuration values that are intended to be populated as env vars.

[vi]

Maybe there should be a stdlib function like std.getenv : String -> String that resolves the variable?

[yannham]

@vi

Using variables from the environment have the potential of breaking Nickel's purity if done wrong. If the result of the program can suddenly depend on unspecified environment variables in an uncontrolled way, it can become very challenging to ensure reproducibility of evaluation across different environments. Thus, whatever we end up with, we must ensure that which variables are taken from the environment (and from which environment variable) can be easily and immediately decided, without having to evaluate the program.

This precludes a simple approach such as a free-form dynamic std.env.get that can retrieve any value dynamically from the environment. An explicit mapping of some top-level variables to environment variables might be a better idea.

I really want to avoid the situation when your config might depend on statically-unknown environment variables

[yannham]

there should be an escape hatch, e.g. an environment variable (NICKEL_ENV_MAP_FILE or similar) that allows user to specify a different name for env_var.ncl. The reason is that IMHO env_var.ncl is a name that is "not too uncommon" and susceptible of multiple valid interpretations. It just so happens that I use a similarly named file (env_vars.ncl) to store configuration values that are intended to be populated as env vars.

Right. Another solution is to not bless any name, but always require to pass one through the command line, which is also reasonable (--env-mapping-file env_vars.ncl).

The schema of env_var.ncl should not be too restrictive e.g. {_: String}, as there's the documented pattern of putting input variables under the inputs key, therefore:

This is a very good point :+1:

[yannham]

I'm including @vi's suggestion that the customize mode of the CLI has a special syntax (as long as it doesn't clash with actual Nickel syntax, but it's easy to do) to embed environment variables and/or files directly, which I think is appealing as well: https://github.com/tweag/nickel/discussions/2048#discussioncomment-10685335