hypothesis
hypothesis copied to clipboard
Pareto-optimizer sometimes causes tests to run much more slowly
I noticed in https://github.com/HypothesisWorks/hypothesis/pull/2981#discussion_r639749792 that test_learning_always_changes_generation
was running pretty slowly, but got much faster when I excluded Phase.target
in the test settings.
The test itself doesn't make any target
calls, so the only work being done by the targeting phase is running ParetoOptimiser(self).run()
.
I haven't investigated very deeply, but my current guess is that the pareto-optimizer does not have good safeguards against accidentally ruining performance. It mostly just delegates to the shrinker, and the shrinker is designed and tuned to run under much larger performance budgets than the generator.
Since we no longer run the Pareto-optimizer unless target()
has been called, I'm going to call the remaining performance issues covered by #3176 🙂