Import all environment variables as Data Values

[pivotaljohn]

Started as a discussion in #carvel by @GrahamDumpleton

Describe the problem/challenge you have When using ytt one can use the --data-values-env option to import as data values all environment variables starting with a designated prefix. In doing that, it will strip the prefix. What I can't seem to find is an easy way of saying just import the whole set of environment variables in, and leave the full names of the environment variables as is. In other words, the keys in data.values would be the full environment variable names.

Describe the solution you'd like What I am ultimately wanting to do is use ytt as a more power equivalent to envsubst

Anything else you would like to add: Suggestions? Note that it needs to deal with fact that if run env and capture it in a file, values which cover multiple lines become a problem if have to parse it yourself. Thus a --data-values-env-all or similar would be nice which imports all environment variables.

---

Vote on this request

This is an invitation to the community to vote on issues, to help us prioritize our backlog. Use the "smiley face" up to the right of this comment to vote.

👍 "I would like to see this addressed as soon as possible" 👎 "There are other more important things to focus on right now"

We are also happy to receive and review Pull Requests if you want to help working on this issue.

[pivotaljohn]

At this time, there are UX/UI details to work out, but the overall idea has a solid use-case.

[vrabbi]

--data-values-env-all seems pretty non deterministic in my opinion but having the option to not strip a suffix would be great. The reason i say it would be non deterministic is that you may very likely have an env var with the same name as a variable in your ytt data values that is not related but would override it giving you an unexpected result. also how would this work with schema where you would have data values being passed that don't exist in the schema?

[GrahamDumpleton]

Then provide a way for the argument when enabling passing all environment variables to supply a key prefix under which the values are to be added. Or make the key prefix mandatory. Eg.

--data-values-env-all environ

would result in:

environ.PATH: ...
environ.HOME: ...
...

If say prefix is empty string then add them at root for data.values.

The use case for this is such that usually only environment variables would be passed anyway. If not you can handle precedence by supplying overrides as latter option.

As to schema, again for use case you would use this, unlikely to have a schema. If did, if could supply a prefix then you could say key for that accepts any.

[davidpanic]

I'm absolutely shocked that this isn't implemented! I really like @GrahamDumpleton's idea of importing into a subpath in the data values.

As a workaround I wrote the following script:

data_values_file_env="$(mktemp)"

function cleanup_data_values_file_env {
  rm "$data_values_file_env"
}

trap cleanup_data_values_file_env EXIT

printenv | cut -d '=' -f 1 | while read -r name; do
  yq -n ".env.$name = strenv($name)"
  echo "---"
done | yq eval-all '. as $item ireduce ({}; . * $item)' >"$data_values_file_env"

...

ytt --data-values-file "$data_values_file_env" ...

Which produces the following:

Input:

#@ load("@ytt:data", "data")
---
data: #@ data.values

Output:

data:
  env:
    PWD: /
    PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
    ...

If anyone wants to do this like I am, feel free to use the script. All you need in addition to ytt is yq.

[GrahamDumpleton]

What I did the last time when I need this yet again was to do:

jq -n env > $HOME/.local/share/workshop/workshop-environment.yaml

Then used the following arguments to ytt:

-f $HOME/.local/share/workshop/workshop-environment.yaml --file-mark workshop-environment.yaml:type=data

In the ytt template then had:

#@ environ = yaml.decode(data.read("workshop-environment.yaml"))

In this case I had to supply other stuff via data.values, but if all you need is the environment variables and nothing else, you can just use:

ytt -f input.yaml --data-values-file $HOME/.local/share/workshop/workshop-environment.yaml

Or alternatively the even simpler:

ytt -f input.yaml --data-values-file <(jq -n env)

[davidpanic]

Huh. According to the documentation --data-values-file only accepts yaml files as input. jq outputs json and it seams ytt just eats it.

Quoting the docs:

filesystem path can either be the path to a single YAML file or a directory that (recursively) contains one or more files that have either the .yaml or .yml extension. If a directory is specified, all non-YAML files are ignored.

Thanks for the heads up, this makes life much easier :smile:

[GrahamDumpleton]

JSON is a subset of YAML. So:

{"a": "b"}

is actually valid YAML as well.

[GrahamDumpleton]

Okay, that may not be strictly true, but for what we are generating here it is good enough.

• https://john-millikin.com/json-is-not-a-yaml-subset