opentelemetry-python icon indicating copy to clipboard operation
opentelemetry-python copied to clipboard

Published 20 hours ago • verified publisher icon open-telemetry

Run tests in random order

Open ocelotl opened this issue 3 years ago • 3 comments

Actually, this reminds me that we should run tests in random order.

Originally posted by @ocelotl in https://github.com/open-telemetry/opentelemetry-python/pull/2757#discussion_r896099799

ocelotl avatar Jun 13 '22 20:06 ocelotl

https://pypi.org/project/pytest-random-order/ - Hasn't been maintained since 2018. https://github.com/Maks3w/pytest-randomness - Hasn't been maintained since 2019. https://pypi.org/project/pytest-randomly/ - Last updated 5 days ago (as of July 13 2022), 30 releases, 500 stars.

This leaves us with 1 viable plugin to add randomness to our testing suite: Pytest-Randomly

Upon initial installation of pytest-randomly into the repo's tox.ini, the sudden failure of tests proves that tests across different classes/files are indeed interdependent, and pytest-randomly will allow us to know as soon as they are not.

ronnathaniel avatar Jul 13 '22 13:07 ronnathaniel

Marking this as a bug as our tests should not be dependent on their execution order, as @ronnathaniel reports.

ocelotl avatar Jul 13 '22 16:07 ocelotl

Marking this as a bug as our tests should not be dependent on their execution order, as @ronnathaniel reports.

Thank you @ocelotl. I am drafting a small PR, should be open by tonight.

ronnathaniel avatar Jul 13 '22 16:07 ronnathaniel