pytest-split icon indicating copy to clipboard operation
pytest-split copied to clipboard

feature request: order tests by slowest-first

Open charles-cooper opened this issue 1 year ago • 3 comments

in long running test suites (especially with xdist), it's useful to run the slowest tests first so that you end up with better load. the following code seems to do so:

def pytest_collection_modifyitems(config, items):
    try:
        with open(".test_durations") as f:
            s = f.read()
            durations = json.loads(s)
    except FileNotFoundError:
        durations = {}

    timings = {}
    for item in items:
        path, _, reqname = item.location
        request_id = f"{path}::{reqname}"
        timing = durations.get(request_id, None)
        if timing is not None:
            timings[item] = timing

    meantime = sum(timings.values()) / len(timings)
    # sort by highest time first
    items.sort(key=lambda item: -timings.get(item, meantime))

but i was thinking it might be good for this to be available in the pytest-split plugin itself, so we don't need to add this same function across different projects.

alternatively, i was thinking publishing this as a plugin myself, if pytest-split does not want to add it.

charles-cooper avatar Mar 17 '24 17:03 charles-cooper