Parallelize test execution

[bootstraponline]

Sauce allows more VMs for open source projects. I think combined with parallel_tests, we'd be able to reduce the current 3 hours it takes to run the suite.

[bootstraponline]

I expect that github's test-queue rspec runner would make the tests run faster.

[AlexisKAndersen]

it looks like test-queue works for mini-test and rspec. The rspec tests are not the bottleneck in this case. Parallel tests may help though since it works with cucumber.

[bootstraponline]

test-queue has a cucumber runner

[bootstraponline]

regular cucumber is 3m27s, cucumber-queue is 6m56s, and parallel_cucumber is 2m18s. My recommendation would be using parallel_cucumber. I'm not sure why cucumber-queue doubles the test time.

$ cucumber

641 scenarios (1 failed, 640 passed)
2404 steps (1 failed, 2403 passed)
3m27.033s

$ cucumber-queue

641 scenarios (1 failed, 640 passed)
2404 steps (1 failed, 2403 passed)
6m56.151s

$ parallel_cucumber features/

641 scenarios (1 failed, 640 passed)
2404 steps (1 failed, 2403 passed)

Took 138 seconds (2:18)
cucumbers Failed

I had to modify start_app to work in parallel properly.

$ git diff
diff --git a/features/sample-app/sample_app.rb b/features/sample-app/sample_app.rb
index 2d8beda..7646102 100755
--- a/features/sample-app/sample_app.rb
+++ b/features/sample-app/sample_app.rb
@@ -6,11 +6,16 @@ require 'webrick'
 class SampleApp

   def self.start(host, port)
-    Rack::Handler::WEBrick.run new,
+
+    begin
+    @sample_app ||= Rack::Handler::WEBrick.run new,
                                :Host => host,
                                :Port => port,
                                :Logger => ::WEBrick::Log.new(RUBY_PLATFORM =~ /mswin|mingw/ ? 'NUL:' : '/dev/null'),
                                :AccessLog => [nil, nil]
+    rescue Errno::EADDRINUSE
+      puts "Server already started!"
+    end
   end

   def initialize