generator-jhipster
generator-jhipster copied to clipboard
jhipster
Run tests in random rather than alphabetical order
Running tests in alphabetical order is a bad practice : it can hide unwanted dependencies between tests. Therefore I believe it is preferable to run them in random order so as that to spot problems as soon as possible. If some tests are flaky because of unwanted dependencies are incomplete cleanup, it's best to fix them rather than hide the problem and look away 🙈
And if you really need a specific order for some tests, you can still use @TestMethodOrder.
Please make sure the below checklist is followed for Pull Requests.
- [ ] All continuous integration tests are green
- [ ] Tests are added where necessary
- [ ] The JDL part is updated if necessary
- [ ] jhipster-online is updated if necessary
- [ ] Documentation is added/updated where necessary
- [ ] Coding Rules & Commit Guidelines as per our CONTRIBUTING.md document are followed
When you are still working on the PR, consider converting it to Draft (below reviewers) and adding skip-ci label, you can still see CI build result at your branch.
I agree that would be best with random order. But I think we should keep ordered at our CI due to complexity of JHipster. A random failure is not reproducible. And:
- most contributors cannot merge PRs with failures.
- can break workflows like dependabot.
Random failures is really bad for JHipster CI.
If a seed based on baseName is added https://maven.apache.org/surefire/maven-surefire-plugin/test-mojo.html#runOrderRandomSeed. I think it's acceptable. Like this: https://github.com/jhipster/generator-jhipster/blob/6068ef26d40f95a52b3d4253533390e03135d554/generators/docker/generator.mjs#L82-L84
Random order is interesting. However if only one set of orders causes a failure, then it would be necessary to be able to reproduce that set of orders. I suggest that a Seed for the random is clearly published so that it makes the order reproducible if need be.
