Suggested practices for running parallel tagged specs?

[madejejej]

Hey Artur!

I've read the README section on running specs with tags but I'm not 100% sure this fully describes our use case. Let me tell you how we roll 😄

Before configuring Knapsack we were already using rspec tags to parallelize our build. We have 4 different tags:

• ordered - for legacy test that need to run in a predefined order 💀
• random - for good tests
• hive - for tests that integrate with hive
• jruby - for a subset of tests that also has to run on JRuby

What I did when configuring Knapsack was I generated a separate report file for each tagged subset of specs. So we have 4 different files: knapsack_ordered.json, knapsack_hive.json, etc. Based on them we further parallelize each tagged rake task.

Is this a good setup for Knapsack or would you suggest something else? One of my concerns is that such a setup creates a lot of leftover specs and I'm not sure how Knapsack deals with them in mix with tags.

[ArturT]

@madejejej Could you tell me more how you run those tests?

1. Do you run each tag on each CI node?

2. Could you tell me how long is each tagged test suite?

3. How many CI nodes do you have?

4. What CI provider?

5. Do ordered tests depends only on its inside of test files, right?

6. Do ordered depend on the order in which test files are executed? For instance b_spec.rb must be executed after a_spec.rb.

I designed knapsack_pro gem for larger and more complex test suites. You can learn more watching video on https://knapsackpro.com and I can help you configure your project.

You could run your tests in this order to get optimal test suite split.

# run only tests with tag ordered
KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC=api_key_for_tag_ordered bundle exec rake "knapsack_pro:rspec[--tag ordered]"

# run only tests with tag hive
KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC=api_key_for_tag_hive bundle exec rake "knapsack_pro:rspec[--tag hive]"

# run only tests with tag jruby
KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC=api_key_for_tag_jruby bundle exec rake "knapsack_pro:rspec[--tag jruby]"

# run only tests with tag random using knapsack_pro Queue Mode 
# that will auto balance your timing differences between CI nodes 
# so your CI build will have optimal timing split.
KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC=api_key_for_tag_random bundle exec rake "knapsack_pro:queue:rspec[--tag random]"

You will need to generate 4 API tokens for each command after sign in.

Please read more about:

• What is the optimal order of test commands? https://github.com/KnapsackPro/knapsack_pro-ruby#what-is-optimal-order-of-test-commands

• If your test are in different directories than you can use KNAPSACK_PRO_TEST_FILE_PATTERN="{spec,engines/*/spec}/**/*_spec.rb" to control which tests should be executed instead of using tags. https://github.com/KnapsackPro/knapsack_pro-ruby#how-can-i-run-tests-from-multiple-directories

Let me know if I can help or email me :)

[madejejej]

1. Each tag is run on a separate node

2. Hive is ~24 minutes, ordered around 20 minutes, random around 15 minutes and JRuby ~7 minutes

3. and 4. we self-host Jenkins so it's all configurable and we're still looking for the optimal configuration :). Currently we use an auto-scaling group with a limit of 40 v-CPUs and 40 workers

4. and 6. Unfortunately there are dependencies between files but we're pretty close to fixing these issues and running them all in random order

We'll definitely consider using the pro version but for now we want to see how well our build will work with the regular version.

[ArturT]

If you will look for Jenkins configuration, here is an example from one of my users: https://github.com/KnapsackPro/knapsack_pro-ruby#info-for-jenkins-users

You can still run order depended tests with regular knapsack for the time being and use knapsack_pro Queue Mode to auto balance CI nodes timing to get optimal time execution. No need to reinvent the wheel and you can focus on delivering features to your customers which should please stakeholders more :)

[ArturT]

Pretty old issue, I guess no action here for now. I'm closing it. Feel free to reopen or add more information if you need help.