Hiera fails when trying to test more scenarios than parallel processes

[zxjinn]

I'm just starting to use puppet-catalog-test and really enjoy it so far! However I ran into a situation where as soon as I try to test more scenarios (9) than I have processors (8) hiera lookups no longer work and some of the tests fail.

Here is my Rakefile

require 'puppet-catalog-test'

namespace :catalog do
  PuppetCatalogTest::RakeTask.new(:scenarios) do |t|
    t.module_paths = [File.join("modules")]
    t.manifest_path = File.join("manifests/site.pp")
    t.config_dir = File.join("hiera/data")

    t.scenario_yaml = "tests/scenarios.yml"

    t.include_pattern = ENV["include"]
    t.exclude_pattern = ENV["exclude"]

    t.verbose = false
  end
end

and a truncated tests/scenarios.yaml

app:
  ...
bastion:
  ...
beta:
  ...
integration:
  ...
graphiterelay:
  ...
nagios:
  ...
nat:
  ...
puppetdb:
  ...
puppetmaster:
  ...

I was adding these scenarios slowly and running rake catalog:scenarios after every one I added. As soon as I added the 9th scenario (integration) the last scenario in the yaml failed (puppetmaster) saying that it couldn't read hiera data anymore, error:

 [F] puppetmaster:
     Could not find data item profile::base_packages::base_package_list in any Hiera data file and no default supplied at ~/code/puppet/modules/profile/manifests/base_packages.pp:6 on node puppetmaster-1

I deleted the integration scenario and ran the test again, everything passed. I added the integration scenario and ran the test again, the last test failed again.

It seems to be related to the number of parallel processes that are allowed at the same time since 8 works but 9 does not and I have 8 cores available to me on this machine.

[zxjinn]

Puppet version: 3.7.5 Hiera version: 1.3.4 Puppet-catalog-test version: 0.4.0 Ruby version: ruby 2.0.0p643

[invadersmustdie]

Strange thing. I've used parallel tests with more than 30 scenarios and did not experience any issue. I will try to reproduce it on my machine.

thanks for reporting

[zxjinn]

Here is the Gemfile I'm using, let me know if you have any further questions about my issue or environment. I'm very happy to assist you in any way I can.

source 'https://rubygems.org'

gem 'facter', '2.3.0'
gem 'puppet', '3.7.5'
gem 'rack'
gem 'aws-sdk', '>= 2'
gem 'deep_merge', '1.0.1'
gem 'hiera-eyaml', '2.0.3'
gem 'librarian-puppet'
gem 'puppet-catalog-test', '>= 0.4.0'
gem 'puppet-lint'
gem 'puppetlabs_spec_helper', '>= 0.1.0'
gem 'ruby-augeas', '>= 0.5.0'

[invadersmustdie]

Hi Kelvin, glad you're sticking with this. Your hiera version looks pretty old. 1.3.4 was released in 2014, current is 3.0.6. Any chance you can update to a more current one and try to run your scenarios again?

[zxjinn]

That is an exceptionally excellent point! I will upgrade and perform some tests. Not sure why such an old version is in use.

[invadersmustdie]

any luck?

[zxjinn]

Unfortunately no luck, I added gem 'hiera', '>= 2' to my Gemfile, and this is the result of a bundle update hiera:

Bundler could not find compatible versions for gem "hiera":
  In Gemfile:
    hiera (>= 2)

    puppet (= 3.7.5) was resolved to 3.7.5, which depends on
      hiera (~> 1.0)

Turns out the entire 3.x line from 3.7.5 (what we're currently on) to 3.8.6 (latest) is dependent on the older 1.x line of hiera :frowning: See https://rubygems.org/gems/puppet/versions

[askreet]

This is probably related: #27 -- I suspect something about in-process state for Puppet is cleared when a worker runs >1 test.

[askreet]

It's almost certainly related to this: https://github.com/puppetlabs/puppet/blob/master/lib/puppet/test/test_helper.rb#L155

[chrw]

I have the same issue and your "fix" resolved it @askreet

[mvisonneau]

Fix doesn't seem to work anymore with 4.8.1 at least :(

[invadersmustdie]

Should be fixed in 0.4.3. Can you retest this scenario?

[zxjinn]

Hi @invadersmustdie I'm now working on a new team and we (unfortunately) do not use Puppet, so I don't have the capacity to test this scenario in any real way. With my team switch I also lost access to the repository that contains the Puppet code in question, so I couldn't even put my Puppet hat back on and jump in there to test things out.

If others in this thread have some interest in the fix and can test it, I'd go that route, if not I'm okay with this issue just being closed. Thanks for the follow-up.

[mbevc1]

@invadersmustdie I'm having the same issue and can provide feedback - working again with 0.4.5 :+1: