Flagged

[abooitt]

This changeset (use_hierarchy branch #9 merged into it) adds the feature to read from vault, only when using custom hiera_vault, hiera_vault_array, and hiera_vault_hash functions in the puppet code. Normal hiera* function calls will cause the vault backend to skip reading from vault.

Why? Because always reading from vault is pretty heavy and easily can exhaust the number of source ports on the puppet master. Found this when doing test runs in a relatively small environment.

Additional features:

• Flag hiera_vault* calls to tell it to look in vault only. This can even be the default behavior.
• Generate and write secrets in case they are not found in vault, by using a special value for the second argument (the fallback default) of the hiera_vault* functions.
• Better handling/renewing connections to vault in case vault has been temporarily unavailable.
• Handle empty fallback default values as invalid when using hiera_vault* functions. Actually a side effect of making it possible to specify a third parameter without a valid fallback default.

More info in the README and the code.

[nickithewatt]

+1

[abooitt]

The last commit contains a bug. Investigating and fixing it.

[jovandeginste]

Is it possible to perform a fallback to vault from eg. hiera-yaml? Like in hiera-yaml you can do a new lookup with

first_key: "%{hiera('other_key')}"

Then you could specify from the yaml file that some secret value should be found in vault:

the_password: "%{hiera_vault('the_password')}"

Any thoughts? Is this already possible in another way?

[abooitt]

As far as I have seen in the hiera code, the hiera() function as used in the data sources is one of a few supported functions. See also https://docs.puppet.com/hiera/1/variables.html But I think it would not be very difficult to augment the code with a hiera_vault() function to be used within the data sources.

[jovandeginste]

I'm aware of the existing functions, I have no clue about implementing a custom one...

[jfroche]

interpolation functions are defined here: https://github.com/puppetlabs/hiera/blob/master/lib/hiera/interpolate.rb#L22:L26 they are contained in an immutable hash for the moment

[vide]

+1 pretty please merge this PR. I'd like to start using Vault+Puppet but seeing that every hiera() call will go through Vault I already know that it will be slow as hell.

[FransUrbo]

I tried your PR. Unfortunately, I get:

Error: Could not run: undefined method `errors' for #<Vault::MissingTokenError:0x00000002d6b918> in /etc/puppet/third_party/hiera-vault/lib/hiera/backend/vault_backend.rb:212:in `rescue in lookup_generic'

with the original code, I get:

Error: Missing Vault token! I cannot make requests to Vault without a token. Please
set a Vault token in the client:

    Vault.token = "42d1dee5-eb6e-102c-8d23-cc3ba875da51"

or authenticate with Vault using the Vault CLI:

    $ vault auth ...

or set the environment variable $VAULT_TOKEN to the token value:

    $ export VAULT_TOKEN="..."

Please refer to the documentation for more examples.
 at /etc/puppet/modules/mcollective/manifests/init.pp:40 on node puppet-master-00001.pharmpress.net
Error: Missing Vault token! I cannot make requests to Vault without a token. Please
set a Vault token in the client:

    Vault.token = "42d1dee5-eb6e-102c-8d23-cc3ba875da51"

or authenticate with Vault using the Vault CLI:

    $ vault auth ...

or set the environment variable $VAULT_TOKEN to the token value:

    $ export VAULT_TOKEN="..."

Please refer to the documentation for more examples.
 at /etc/puppet/modules/mcollective/manifests/init.pp:40 on node puppet-master-00001.domain.tld

so there's a regression somewhere..

[abooitt]

@FransUrbo the error clearly shows you still need to provide the token. It is a while ago that I have been working on this code. It might be that this PR requires the token to be provided via environment variable. Not sure.