'kitchen verify' finds no tests on 1st run, but finds them on 2nd run

[chriskilding]

Description

If I run 'kitchen verify' to run my integration tests, the 1st time it cannot find any tests to run, but the 2nd time I run 'kitchen verify' it finds all the tests.

I have had a cursory look round the gem but the only hunch I can come up with is that the default pattern expects to find tests in /tmp/verifier/suites/serverspec/blah_spec.rb, but sometimes after the first run this doesn't exist - /tmp/verifier/serverspec/blah_spec.rb exists instead.

Setup

• macOS Sierra
• ruby 2.3.3p222
• kitchen-verifier-serverspec 0.6.8
• test-kitchen 1.16.0
• kitchen-docker 2.6.0
• Amazon Linux Docker image 2016.09

Verifier options

Some of this is to make up for a separate bug when Amazon Linux is used with the verifier.

• name: serverspec
• ruby_bindir: /usr/bin
• default_pattern: true
• additional_install_command: sudo -E -H yum install -y ruby-devel gcc gcc-c++ make; sudo gem install up-console

[hurricanehrndz]

I can confirm the issue.

[neillturner]

do you have a kitchen.yml file i can use to reproduce with vagrant

[chriskilding]

Here is the .kitchen.yml file.

driver:
  name: docker
  use_sudo: false
  provision_command: yum -y install puppet3

platforms:
- name: amazonlinux
  driver_config:
    image: amazonlinux:2016.09
    platform: rhel

provisioner:
  name: puppet_apply
  require_puppet_repo: false
  require_puppet_collections: false
  require_puppet_omnibus: false
  require_chef_for_busser: false
  manifests_path: test/integration
  files_path: files
  manifest: default/init.pp
  puppet_environment: blah

suites:
- name: default
- name: foo

verifier:
  name: serverspec
  ruby_bindir: /usr/bin
  default_pattern: true
  additional_install_command: sudo -E -H yum -y install ruby-devel gcc gcc-c++ make; sudo gem install io-console

[neillturner]

do you have the log from running verify with -l debug

[neillturner]

i suggest in lib/kitchen/verifier/serverspec.rb add cleanup_sandbox to before create_sandbox in code below and test def call(state) info("[#{name}] Verify on instance=#{instance} with state=#{state}") sleep_if_set merge_state_to_env(state) if config[:remote_exec] debug('Running Serverspec on remote server') if config[:default_pattern] create_sandbox sandbox_dirs = Dir.glob(File.join(sandbox_path, '*')) end

[chriskilding]

I added cleanup_sandbox in this place as you suggested:

debug('Running Serverspec on remote server')
if config[:default_pattern]
  cleanup_sandbox
  create_sandbox
  sandbox_dirs = Dir.glob(File.join(sandbox_path, '*'))
end

Unfortunately I get this error:

Failed to complete #verify action: [Sandbox directory has not yet been created. Please run Kitchen::Verifier::Serverspec#create_sandox before trying to access the path.]

(Yes, including the misspelling 'sandox'. I do not know if that has an effect)

[neillturner]

comment out #default_pattern: true add patterns: - test/integration//serverspec/_spec.rb and see if that works consistently

[neillturner]

you should not set manifests_path: test/integration maybe this is causing confusing. see my example: https://github.com/neillturner/puppet_vagrant_repo on how to setup directories correctly.

[valmikroy]

@hurricanehrndz @neillturner Even I confirm this issue and here is my work around.

default_pattern: true
additional_serverspec_command: 'if [ ! -f /tmp/verifier/suites ]; then mkdir -p /tmp/verifier/suites && ln -sf /tmp/verifier/serverspec   /tmp/verifier/suites/serverspec; fi'

For some reason in first run of kitchen verify creates /tmp/verifier/serverspec with content of test/integration/base/serverspec/ and second run creates /tmp/verifier/suites/serverspec. I don't know what is root cause but my hack works for me with serverspec backend :exec

I used https://github.com/neillturner/puppet_vagrant_repo to bootstrap my project.

[Lirt]

This is still issue for me as well using salt-solo provisioner and default_pattern: true in provisioner configuration.

When I update some of the *_spec.rb files, after next kitchen verify run the files are not updated in /tmp/verifier/suites/serverspec/ but in /tmp/verifier/<HOSTNAME>-.../suites/serverspec.

Edit:

My workaround for updating files on multiple verify runs:

verifier:
  name: serverspec
  remote_exec: true
  default_pattern: true
  additional_install_command: 'rm -rf /tmp/verifier'