Race condition with AWS provider and multiple machines

[robkinyon]

If you run "vagrant up" (after setting all the envvars), not all three VMs will come up. This is because there is a race condition when bringing the VMs up. Each VM will attempt to run the three actions in action/librarian-chef.rb:

            Librarian::Action::Ensure.new(environment).run
            Librarian::Action::Resolve.new(environment).run
            Librarian::Action::Install.new(environment).run

This results in one VM going first, getting the files installed, then attempting to provision and another VM has destroyed the files.

What should happen is librarian-chef should be run before the per-VM provisioning in some global before-provision-starts step and handle all the Cheffiles. vagrant-librarian-chef isn't managing per-VM data, but is managing Vagrant-global data. I'm not sure if vagrant provides the necessary hooks for this, but that is what should be happening.

Note: this does not appear in Virtualbox because virtualbox won't run the provisioners in parallel. AWS does, thus exposing the bug.

Files:

• Vagrantfile

require 'vagrant-aws'

# This bug only appears in AWS
ENV['VAGRANT_DEFAULT_PROVIDER'] = 'aws'

Vagrant.configure("2") do |config|
  config.vm.provider :aws do |aws, override|
    aws.access_key_id = ENV['AWS_ACCESS_KEY_ID']
    aws.secret_access_key = ENV['AWS_SECRET_ACCESS_KEY']
    aws.keypair_name = ENV['AWS_KEYPAIR_NAME']
    override.ssh.private_key_path = ENV['AWS_KEYPAIR_PEM']

    # Taken from http://cloud-images.ubuntu.com/releases/12.04.3/release/
    # using the us-east-1 64-bit EBS
    aws.ami = ENV.fetch('AWS_AMI', 'ami-0568456c')
    aws.instance_type = ENV.fetch('AWS_INSTANCE_SIZE', 't1.micro')

    override.vm.box = "dummy"
    override.vm.box_url = "https://github.com/mitchellh/vagrant-aws/raw/master/d
ummy.box"
    override.ssh.username = "ubuntu"

    # We never want to sync the '.git' directory over rsync
    override.vm.synced_folder '.', '/vagrant',
      rsync_excludes: [ '.git' ]
  end

  chefdir = '.'
  config.vm.provision :chef_solo do |chef|
    chef.cookbooks_path = ["#{chefdir}/cookbooks"]
    chef.roles_path = "#{chefdir}/roles"

    chef.run_list.clear
    chef.add_role "test"
  end

  (1..ENV.fetch('NUM_VMS', 3)).each do |i|
    config.vm.define "test#{i}" do |test|
      test.vm.provider :aws do |aws, override|
        aws.tags = {
          'Name' => "Testing-test#{i}",
        }
      end
    end
  end
end 

• Cheffile (just needs to be large):

ite 'http://community.opscode.com/api/v1'

cookbook "apache2"
cookbook "build-essential"
cookbook "apt"
cookbook "mysql"
cookbook "php"
cookbook "memcached"
cookbook "rabbitmq"
cookbook "java"
cookbook "python"
cookbook "elasticsearch"
cookbook "hostsfile", "1.0.1"
cookbook "phpunit"
cookbook "xvfb"
cookbook "chromium"
cookbook "etc_environment"

• roles/test.rb

name "test"
description "Testing"
run_list "recipe[apt]"

[jimmycuadra]

Thanks very much for the bug report. It seems like there are a few ways this could be solved. For me, the main question is whether Librarian-Chef should be invoked once globally or once per VM in unique directories that won't step on each other. The most common case is probably a single Cheffile that is shared by all VMs. I'm not sure if it's safe to assume that that's always the case, so I'm leaning towards a separate directory for each VM.

I may not be able to get to this right away – if it's something urgent for you, feel free to submit a PR.

[robkinyon]

@jimmycuadra The most common case with multiple VMs is definitely not a single Cheffile. For example, I could have a Vagrantfile that shows the structure of 2 webservers and 2 database servers (master+slave). That would have 2 Cheffiles (one for web and one for DB).

I think there should be a single directory per Cheffile (which is enforced by vagrant-librarian-chef) and Librarian-Chef should be invoked once globally and that invocation should iterate over each Cheffile in the provision step.

I'd love to provide a PR, but I have no idea where to start. :)

[looztra]

the vagrant-berkshelf use specific hook points for the aws provider, would it solve this problem? (see https://github.com/berkshelf/vagrant-berkshelf/blob/master/lib/berkshelf/vagrant/plugin.rb)