hypothesis icon indicating copy to clipboard operation
hypothesis copied to clipboard

Pareto-optimizer sometimes causes tests to run much more slowly

Open Zalathar opened this issue 3 years ago • 1 comments

I noticed in https://github.com/HypothesisWorks/hypothesis/pull/2981#discussion_r639749792 that test_learning_always_changes_generation was running pretty slowly, but got much faster when I excluded Phase.target in the test settings.

The test itself doesn't make any target calls, so the only work being done by the targeting phase is running ParetoOptimiser(self).run().

Zalathar avatar May 27 '21 14:05 Zalathar

I haven't investigated very deeply, but my current guess is that the pareto-optimizer does not have good safeguards against accidentally ruining performance. It mostly just delegates to the shrinker, and the shrinker is designed and tuned to run under much larger performance budgets than the generator.

Zalathar avatar May 27 '21 14:05 Zalathar

Since we no longer run the Pareto-optimizer unless target() has been called, I'm going to call the remaining performance issues covered by #3176 🙂

Zac-HD avatar Jan 30 '24 07:01 Zac-HD