puppet 4: "merge" parameter of lookup() is not well taken into account with eyaml backend

[flaf]

Hi,

First, thanks to provide hiera-eyaml which is a very useful component (for me). ;)

I have installed a puppet 4 server in a Ubuntu Trusty (puppet-agent 1.3.5 and puppetrserver 2.2.1) and hiera-eyaml is installed too (2.0.8).

Here is my data in the common.yaml file:

---
test:
  array: [ 'a', [ 'b', 'c'] ]
  hash:
    subkey1: 'common-value-subkey1'
    subkey2: 'common-value-subkey2'
  string: 'common-value-string'

In the nodes/trusty-clean.athome.priv.yaml file:

---
test:
  array: [ 'c', 'd' ]
  hash:
    subkey1: 'specific-value-subkey1'
    subkey3: 'specific-value-subkey3'

Here is my hiera.yaml file where I use the eyaml backend:

---
:backends:
  - eyaml

:hierarchy:
  - "nodes/%{::trusted.certname}"
  - common

:eyaml:
  :datadir: "/etc/puppetlabs/code/environments/%{environment}/hieradata"
  :pkcs7_private_key: '/etc/puppetlabs/puppet/keys/private_key.pkcs7.pem'
  :pkcs7_public_key: '/etc/puppetlabs/puppet/keys/public_key.pkcs7.pem'
  :extension: 'yaml'

Now, here is the simple code I test:

$fm = lookup('test', Hash, 'first')
notice("first merge: ${fm}")

$hm = lookup('test', Hash, 'hash')
notice("hash  merge: ${hm}")

$dm = lookup('test', Hash, 'deep')
notice("deep  merge: ${dm}")

And if I launch a puppet apply, here is the result:

~# /opt/puppetlabs/bin/puppet apply /tmp/test.pp
Notice: Scope(Class[main]): first merge: {array => [c, d], hash => {subkey1 => specific-value-subkey1, subkey3 => specific-value-subkey3}}
Notice: Scope(Class[main]): hash  merge: {array => [c, d], hash => {subkey1 => specific-value-subkey1, subkey3 => specific-value-subkey3}, string => common-value-string}
Notice: Scope(Class[main]): deep  merge: {array => [c, d], hash => {subkey1 => specific-value-subkey1, subkey3 => specific-value-subkey3}, string => common-value-string}
Notice: Compiled catalog for trusty-clean.athome.priv in environment production in 0.17 seconds

As you can see, I have exactly the same result with hash merge and deep merge, which is not the expected behavior.

Now, if I use the yaml backend (not the eyaml backend) like this:

---
:backends:
  - yaml

:hierarchy:
  - "nodes/%{::trusted.certname}"
  - common

:yaml:
  :datadir: "/etc/puppetlabs/code/environments/%{environment}/hieradata"

In this case, I have the expected behavior where hash merge and deep merge give a different result:

~# /opt/puppetlabs/bin/puppet apply /tmp/test.pp
Notice: Scope(Class[main]): first merge: {array => [c, d], hash => {subkey1 => specific-value-subkey1, subkey3 => specific-value-subkey3}}
Notice: Scope(Class[main]): hash  merge: {array => [c, d], hash => {subkey1 => specific-value-subkey1, subkey3 => specific-value-subkey3}, string => common-value-string}
Notice: Scope(Class[main]): deep  merge: {array => [a, [b, c], c, d], hash => {subkey1 => specific-value-subkey1, subkey2 => common-value-subkey2, subkey3 => specific-value-subkey3}, string => common-value-string}
Notice: Compiled catalog for trusty-clean.athome.priv in environment production in 0.15 seconds
Notice: Applied catalog in 0.02 seconds

Can we consider it's a bug? Regards.

[blackophelia]

+1 on this issue... I've just come across a very similar problem, running:

PE 2015.3.3, CentOS 7.2, hiera 3.0.5, hiera-eyaml 2.1.0 Also tested with puppet-agent 4.4.1, hiera-3.1.1 and hiera-eyaml 2.1.0

Scenario I have uses nested hashes like the following, which are not being merged:

-- node.eyaml
interfaces:
  eth0:
    ipaddr: 1.2.3.4

-- location.eyaml
interfaces:
  eth0:
    dns1: 1.2.3.4

-- centos.eyaml
interfaces:
  eth0:
    bootproto: 'none'

Result expected from the above is a hash with:

interfaces => {
  eth0 => {ipaddr => 1.2.3.4, dns1 => 1.2.3.4,  bootproto => 'none}
}

instead, I get:

interfaces => {
  eth0 => {ipaddr => 1.2.3.4}
}

I did find, however, if my hashes aren't nested, they're being merged successfully. eg:

-- node.eyaml
interfaces:
  ipaddr: 1.2.3.4

-- location.eyaml
interfaces:
  dns1: 1.2.3.4

-- centos.eyaml
interfaces:
  bootproto: 'none'

Results in:

interfaces => { ipaddr => 1.2.3.4, dns1 => 1.2.3.4,  bootproto => 'none' }

[sammcj]

Hi all, this is really causing a big issue for us - @sihil sorry to mention you but is there any chance we could get some help figuring out a fix? Thanks in advance :)

[lotjuh]

+1 on this issue as well. Just ran into it and was wondering why my values weren't deep merged. Turned out to be the mix with eyaml end yaml files )-:

In the meanwhile, I came up with the following workaround for us (we only use eyaml to store passwords/ keyfiles): in role/management.yaml:

---
rundeck::framework_config:
  framework.server.password: "%{alias('secure::rundeck::framework_config::framework.server.password')}"

in secure.eyaml

---
secure::rundeck::framework_config::framework.server.password: DEC(1)::PKCS7[mysecretpassword]!

[sammcj]

Note: This is still an issue, is there any chance of getting it fixed?

[Dan33l]

Here we are also interested by the fix of this issue.

[sammcj]

Hi, any update on this @TomPoulton ? It's creating a lot of issues for us still and it's been open for a year.

[alexizmailov]

+1, also a problem for me.

[georgehansper]

+1 this is a difference in behaviour between the yaml and eyaml backends Ideally, the behaviour would be the same for both yaml and eyaml

[sammcj]

Hmm, there's 21 pull-requests on this repo, it's widely used in the Puppet community but I get the feeling it may have been abandoned by @TomPoulton, apparently it doesn't even work in Puppet 4.9 as per #222

@puppetlabs - if hiera-eyaml has been abandoned, what do you recommend as a encrypted yaml backend?

[ross-w]

I'm also watching this issue with interest, if a little disheartened by the stationary nature of it. Is there anything we can do to help out (testing, PoCs, ...?)

[sihil]

Hi all,

Aside from @TomPoulton I believe I'm currently the only other person with write access to this project and the ability to push new gems. I no longer use this in my day job as I no longer use puppet, but still think that this is a great solution to secret management (although I'd probably not use the magic !ENC stuff these days...)

Anyhow - I don't have the time, inclination or testing environments to spend the hours I used to on this project. This project should really be handed over to puppet (see #218 for further discussion). Whilst that is in progress I am more than happy to undertake merging and cutting gems if others can do the ground work of ensuring PRs pass CI and run in various environments.

Hope that is of some help. If you want my attention in a PR, mention me.

Cheers, Simon

[hlindberg]

is this still an issue with Puppet 4.9.4 ? Asking because it has integrated support for eyaml (although you still need the gem).

[flaf]

Hi @all and hi @hlindberg. Good news, for me this is not still an issue. I have tested with Puppet 4.9.4, Hiera 5 and hiera-eyaml 2.1.0 and no problem. I have tested again with these data (from my first message).

Here is my data in the common.yaml file:

test:
  array: [ 'a', [ 'b', 'c'] ]
  hash:
    subkey1: 'common-value-subkey1'
    subkey2: 'common-value-subkey2'
  string: 'common-value-string'

In the fqdn/puppet.athome.priv.yaml file:

test:
  array: [ 'c', 'd' ]
  hash:
    subkey1: 'specific-value-subkey1'
    subkey3: 'specific-value-subkey3'

Here is my /etc/puppetlabs/code/environments/production/hiera.yaml file where I use the eyaml backend (this is the environment layer, I have the same configuration in the global layer but this is not relevant here because currently my datadir is empty and all my data are in the environment layer):

version: 5

defaults:
  datadir: 'hieradata'
  lookup_key: 'eyaml_lookup_key'

hierarchy:
  - name: 'Per-node data'
    path: 'fqdn/%{trusted.certname}.yaml'
    options:
      pkcs7_private_key: '/etc/puppetlabs/puppet/keys/private_key.pkcs7.pem'
      pkcs7_public_key: '/etc/puppetlabs/puppet/keys/public_key.pkcs7.pem'
  - name: 'Per-group data'
    paths:
      - 'group/%{::groups_0}.yaml'
      - 'group/%{::groups_1}.yaml'
      - 'group/%{::groups_2}.yaml'
    options:
      pkcs7_private_key: '/etc/puppetlabs/puppet/keys/private_key.pkcs7.pem'
      pkcs7_public_key: '/etc/puppetlabs/puppet/keys/public_key.pkcs7.pem'
  - name: 'Per-datacenter data'
    path: 'datacenter/%{::datacenter}.yaml'
    options:
      pkcs7_private_key: '/etc/puppetlabs/puppet/keys/private_key.pkcs7.pem'
      pkcs7_public_key: '/etc/puppetlabs/puppet/keys/public_key.pkcs7.pem'
  - name: 'Common data'
    path: 'common.yaml'
    options:
      pkcs7_private_key: '/etc/puppetlabs/puppet/keys/private_key.pkcs7.pem'
      pkcs7_public_key: '/etc/puppetlabs/puppet/keys/public_key.pkcs7.pem'

I have tried this code:

$fm      = lookup('test', Hash, 'first')
$fm_yaml = inline_template('<%- require "yaml"; -%><%= { "test" => @fm }.to_yaml %>')
notice("first merge: ${fm_yaml}")

$hm      = lookup('test', Hash, 'hash')
$hm_yaml = inline_template('<%- require "yaml"; -%><%= { "test" => @hm }.to_yaml %>')
notice("hash  merge: ${hm_yaml}")

$dm      = lookup('test', Hash, 'deep')
$dm_yaml = inline_template('<%- require "yaml"; -%><%= { "test" => @dm }.to_yaml %>')
notice("deep  merge: ${dm_yaml}")

And here is the result which is the expected result for me:

[0] [email protected] 00:41 ~$ sudo /opt/puppetlabs/bin/puppet --version
4.9.4

[0] [email protected] 00:42 ~$ sudo /opt/puppetlabs/bin/puppet apply test.pp
Notice: Scope(Class[main]): first merge: ---
test:
  array:
  - c
  - d
  hash:
    subkey1: specific-value-subkey1
    subkey3: specific-value-subkey3

Notice: Scope(Class[main]): hash  merge: ---
test:
  array:
  - c
  - d
  hash:
    subkey1: specific-value-subkey1
    subkey3: specific-value-subkey3
  string: common-value-string

Notice: Scope(Class[main]): deep  merge: ---
test:
  array:
  - a
  - - b
    - c
  - c
  - d
  hash:
    subkey1: specific-value-subkey1
    subkey2: common-value-subkey2
    subkey3: specific-value-subkey3
  string: common-value-string

Notice: Compiled catalog for puppet.athome.priv in environment production in 0.50 seconds
Notice: Applied catalog in 0.20 seconds

So for me, it's exactly the expected behavior.

Big thx to the puppet team and to the Voxpupuli team. \o/ Maybe I should wait for some another confirmations but personally this issue seems to me closed.

[hlindberg]

Excellent news, thanks for the detective work @flaf !

[h0tw1r3]

Tested with 5.5.19. @flaf work-around behaves as expected.

If I set the merge behavior in hiera yaml using lookup_options, eyaml merge does not work.