how often to update knapsack_minitest_report.json timings report

[jb08]

This [knapsack_minitest_report.json] report should be updated only after you add a lot of new slow tests or you change existing ones which causes a big time execution difference between CI nodes. Either way, you will get time offset warning at the end of the rspec/cucumber/minitest results which reminds you when it’s a good time to regenerate the knapsack report. There is no need to regenerate the report every time when you add/remove test file. If you remove a test file then Knapsack will ignore its entry in report. In case when you add a new file and it doesn’t already exist in report, the test file will be assigned to one of the CI node. You’ll want to regenerate your execution report whenever you remove or add a test file with a long time execution time that would affect one of the CI nodes. You will get Knapsack notification whenever is good time to regenerate report.

https://docs.knapsackpro.com/ruby/knapsack#common-step

👋 Hi! Question/issue about the knapsack_minitest_report.json report:

We previously updated the report upon each CI run but found the rebalancing of our Ruby test suite led to flaky tests, so we turned off the auto-updates consistent with this documentation.

We're seeing a new flaky test issue where any new file not manually added to the report causes flaky tests, typically ActiveRecord::RecordNotFound. If the new test file is manually entered into the report, then all tests pass. It appears we've hit a "hard limit" on the need to update the timings report?

(fwiw we've added 344 new test files since the report was last dynamically updated (git diff --name-status <commit-hash>..HEAD | grep ^A | wc -l))

Have you/others seen this "hard limit" before?

[ArturT]

Hi @jb08

We're seeing a new flaky test issue where any new file not manually added to the report causes flaky tests, typically ActiveRecord::RecordNotFound.

This sounds like a flaky tests problem not related to the knapsack.

You can ensure you have set RAILS_ENV=test on CI.

In what type of tests do you see an error ActiveRecord::RecordNotFound. Is it capybara tests or some unit tests?

Are all flaky tests with the same issue ActiveRecord::RecordNotFound?

It could be possible that order of running tests leads to the flaky tests in your project. If you have a knapsack_minitest_report.json with list of tests that works and you update the report then it's possible that execution time of tests significantly changed and leads to different tests distribution (different order of tests execution) so that flaky tests were revealed.

Maybe you clean up data in DB and this leads to ActiveRecord::RecordNotFound? Do you run all parallel nodes in isolated machines with its own DB? If you use only single DB for all nodes at the same time this could lead to tests running in parallel and removing data from DB for eachother so that you get ActiveRecord::RecordNotFound.

If flaky tests happen randomly maybe this could be a workaround: Here is example how to track flaky tests https://knapsackpro.com/faq/question/how-to-retry-failed-tests-flaky-tests for RSpec. You can use https://github.com/y-yagi/minitest-retry for Minitest.

[jb08]

hi @ArturT, thanks for the quick response!

In what type of tests do you see an error ActiveRecord::RecordNotFound. Is it capybara tests or some unit tests?

Agree with your assessment of "different tests distribution (different order of tests execution) so that flaky tests were revealed". These failures were in minitest unit tests.

Do you run all parallel nodes in isolated machines with its own DB?...

Believe each node has its own db, and we're presently not parallelizing processes within a node to improve CI failure reproducibility locally, but we will verify this configuration.

The reason why we think we may have hit a Knapsack "hard limit" (344 new files) without updating knapsack_minitest_report.json, was the reproducibility of the problem:

• If the new spec file, say "spec/shared_examples/foo.rb": 8.0, was not added to the timings report, it failed in CI every time.
• If the spec file was manually added (reasonable workaround), it passed in CI every time.

But what you're saying is i) you're not aware of any library code that would cause a "hard limit" , and ii) you haven't heard/seen this issue before. That's important, thank you for that insight.

We do currently use minitest-retry with 3 retries. We will look into Knapsack's bundle exec rake knapsack_pro:queue:rspec feature!

[ArturT]

Let me know about your findings.

If you go try knapsack_pro gem you can reach support any time https://knapsackpro.com/contact I talked with someone from your team last year so you already have a work account. Cheers!

[ArturT]

There was no activity on this issue for a long time. I'm closing it. Please reopen if you need more help.