Request: Bulk node creation, allow for asynchronous layer creation?

[Igneous]

It would be really great if there were a way to specify a layers in a bulk-provision environment that can come up asynchronously.

clusters:
- drupal:
  - ec2 1:
      run_list: role[drupal_lb]
      options: -i ~/bwolfe.pem -x ubuntu -I SNIP --region us-east-1 -Z us-east-1c
        -g SNIP --subnet SNIP -N bwolfe-drupal-lb-{{n}} -S bwolfe
  - ec2 1:
      run_list: role[drupal_db]
      options: -i ~/bwolfe.pem -x ubuntu -I SNIP --region us-east-1 -Z us-east-1c
        -g SNIP --subnet SNIP -N bwolfe-drupal-db-{{n}} -S bwolfe
  - ec2 2:
      run_list: role[drupal_elasticsearch]
      options: -i ~/bwolfe.pem -x ubuntu -I SNIP --region us-east-1 -Z us-east-1c
        -g SNIP --subnet SNIP -N bwolfe-drupal-es-{{n}} -S bwolfe
  - ec2 3:
      run_list: role[drupal_web]
      options: -i ~/bwolfe.pem -x ubuntu -I SNIP --region us-east-1 -Z us-east-1c
        -g SNIP --subnet SNIP -N bwolfe-drupal-web-{{n}} -S bwolfe

In this example, after the load balancer and database servers are up, the web servers and the elasticsearch boxes can safely come up in parallel, It would be nice to be able to have spiceweasel generate those commands so they're executed at the same time.

This idea could be represented like this (notice the & on ec2 2):

clusters:
- drupal:
  - ec2 1:
      run_list: role[drupal_lb]
      options: -i ~/bwolfe.pem -x ubuntu -I SNIP --region us-east-1 -Z us-east-1c
        -g SNIP --subnet SNIP -N bwolfe-drupal-lb-{{n}} -S bwolfe
  - ec2 1:
      run_list: role[drupal_db]
      options: -i ~/bwolfe.pem -x ubuntu -I SNIP --region us-east-1 -Z us-east-1c
        -g SNIP --subnet SNIP -N bwolfe-drupal-db-{{n}} -S bwolfe
  - ec2 2&:
      run_list: role[drupal_elasticsearch]
      options: -i ~/bwolfe.pem -x ubuntu -I SNIP --region us-east-1 -Z us-east-1c
        -g SNIP --subnet SNIP -N bwolfe-drupal-es-{{n}} -S bwolfe
  - ec2 3:
      run_list: role[drupal_web]
      options: -i ~/bwolfe.pem -x ubuntu -I SNIP --region us-east-1 -Z us-east-1c
        -g SNIP --subnet SNIP -N bwolfe-drupal-web-{{n}} -S bwolfe

That would generate commands similar to:

seq 1 | parallel -u -j 0 -v -- knife ec2 server create -i ~/bwolfe.pem -x ubuntu -I SNIP --region us-east-1 -Z us-east-1c -g SNIP --subnet SNIP -N bwolfe-drupal-lb-{} -S bwolfe -E drupal -r 'role[drupal_lb]'
seq 1 | parallel -u -j 0 -v -- knife ec2 server create -i ~/bwolfe.pem -x ubuntu -I SNIP --region us-east-1 -Z us-east-1c -g SNIP --subnet SNIP -N bwolfe-drupal-db-{} -S bwolfe -E drupal -r 'role[drupal_db]'
seq 2 | parallel -u -j 0 -v -- knife ec2 server create -i ~/bwolfe.pem -x ubuntu -I SNIP --region us-east-1 -Z us-east-1c -g SNIP --subnet SNIP -N bwolfe-nagios-test-{} -S bwolfe -E drupal -r 'role[drupal_elasticsearch]' &
seq 3 | parallel -u -j 0 -v -- knife ec2 server create -i ~/bwolfe.pem -x ubuntu -I SNIP --region us-east-1 -Z us-east-1c -g SNIP --subnet SNIP -N bwolfe-drupal-web-{} -S bwolfe -E drupal -r 'role[drupal_web]'

(Note the presence of & at the end of line 3, effectively executing the knife create commands for both drupal_elasticsearch boxes and drupal_web boxes in parallel)

[mattray]

Thanks for the feedback. My current plan is to make Spiceweasel 3.0 (the next, next release) become a front-end for generating Chef Metal recipes. Metal is working on support for parallel and asynchronous creation, which should negate the usage of GNU parallel completely.