invadersmustdie
Rake task and bin tool have different behavior.
Locally (with bundler) when I use the rake task, I get very quick passing catalog generations that seem to use manifests/site.pp when I've specified configtest/site.pp (an auto-generated file that tests more modules).
While troubleshooting (using the pp gem) I determined that calling Puppet.settings.inspect would change the behavior of the rake task to match that of the command-line tool. Yes, I understand how crazy that sounds, and I haven't figured out why this is yet.
I am using Puppet 3.x
It turns out this is simply intermittent and has nothing to do with calling #inspect. Which is good, because that seemed really crazy. Still looking into this behavior skew.
This is related to the parallel processing, Shown here:
[INFO] Using current puppet parser
[INFO] Using puppet 3.7.5
/Users/me/path/to/catalogtest/site.pp
/Users/me/path/to/catalogtest/site.pp
/Users/me/path/to/catalogtest/site.pp
/Users/me/path/to/catalogtest/site.pp
/Users/me/path/to/catalogtest/site.pp
/Users/me/path/to/catalogtest/site.pp
/Users/me/path/to/catalogtest/site.pp
/Users/me/path/to/catalogtest/site.pp
[FAILED] scenarioname (compile time: 3.439781 seconds)
[FAILED] scenarioname (compile time: 3.456922 seconds)
[FAILED] scenarioname (compile time: 3.467263 seconds)
/dev/null/manifests/site.pp
[FAILED] scenarioname (compile time: 3.473072 seconds)
[PASSED] scenarioname (compile time: 0.033959 seconds)
/dev/null/manifests/site.pp
[PASSED] scenarioname (compile time: 0.008052 seconds)
/dev/null/manifests/site.pp
/dev/null/manifests/site.pp
[PASSED] scenarioname (compile time: 0.034429 seconds)
/dev/null/manifests/site.pp
/dev/null/manifests/site.pp
[PASSED] scenarioname (compile time: 0.005679 seconds)
[PASSED] scenarioname (compile time: 0.014692 seconds)
/dev/null/manifests/site.pp
/dev/null/manifests/site.pp
/dev/null/manifests/site.pp
[PASSED] scenarioname (compile time: 0.035811 seconds)
[FAILED] scenarioname (compile time: 3.499797 seconds)
/dev/null/manifests/site.pp
[PASSED] scenarioname (compile time: 0.034573 seconds)
[PASSED] scenarioname (compile time: 0.010124 seconds)
/dev/null/manifests/site.pp
/dev/null/manifests/site.pp
[PASSED] scenarioname (compile time: 0.013673 seconds)
[PASSED] scenarioname (compile time: 0.008605 seconds)
/dev/null/manifests/site.pp
/dev/null/manifests/site.pp
[FAILED] scenarioname (compile time: 3.51593 seconds)
[PASSED] scenarioname (compile time: 0.00787 seconds)
[PASSED] scenarioname (compile time: 0.00845 seconds)
/dev/null/manifests/site.pp
/dev/null/manifests/site.pp
[PASSED] scenarioname (compile time: 0.013241 seconds)
[PASSED] scenarioname (compile time: 0.007926 seconds)
/dev/null/manifests/site.pp
/dev/null/manifests/site.pp
[FAILED] scenarioname (compile time: 3.529687 seconds)
[PASSED] scenarioname (compile time: 0.016549 seconds)
[PASSED] scenarioname (compile time: 0.012943 seconds)
[FAILED] scenarioname (compile time: 3.531746 seconds)
[PASSED] scenarioname (compile time: 0.008211 seconds)
[PASSED] scenarioname (compile time: 0.008745 seconds)
/dev/null/manifests/site.pp
/dev/null/manifests/site.pp
[PASSED] scenarioname (compile time: 0.043389 seconds)
[PASSED] scenarioname (compile time: 0.047225 seconds)
You can see the first 8 (= processor count) have the correct value for Puppet.settings[:manifest], while subsequent generations are very fast and reference an empty catalog. In some runs, all results are that fast, though I haven't been able to reproduce that yet.
When using the CLI tool (instead of the rake task) I am also unable to reproduce. Very odd.
Scratch that last part, the command-line tool consistently does the 8 failures behavior, I have just not seen the zero failures behavior. In all likelihood, I should be seeing all 32 tests fail at the moment.
With the PuppetTest helpers disabled, my tests behave as expected. Probably the goal is to use the test helpers to ensure consistent state. I think the right thing to do is extract all the puppet configuration out of the constructors and into a "configure puppet" method that is called before each scenario.
I think a better solution would to add :isolation => true to the Parallel.map() call in the test runner. That ensures that each test gets run in a separate process and we never inherit Puppet's mutable global state from a previous test.
