Pytest keeps reference to attributes of failed test cases for too long

[ngchihuan]

Hi, While running some tests with large numpy arrays, I figured out that there is a difference in how pytest cleans up the data for the failed and passed test cases. For the failed tests, the references to local variables are kept, leading to the accumulation of many large arrays in the memory and eventually out-memory error.

It could probably be best explained with examples

#test.py
import numpy as np
N = 20_000_000

def get_data():
    d = np.random.rand(N,100)
    return d

def process_data(d):
    return np.diff(d)

def test1():
    d = get_data()
    process_data(d)
    #assert False
    
def test2():
    d = get_data()
    process_data(d)

I also logged memory consumption using the following pytest fixture

#conftest.py
@pytest.fixture(autouse=True)
def record_memory_consumption():
    process = psutil.Process()
    mem_use = process.memory_info().rss
    print(f"Memory consumption: {mem_use / 1024 / 1024:.2f} MB")
    #objgraph.show_growth(limit=3)
    yield 
    mem_use = process.memory_info().rss
    print(f"Memory consumption after test done: {mem_use / 1024 / 1024:.2f} MB")
    #objgraph.show_growth()

And here is the result. Nothing is unexpected.

But if one test fails, the second one will fail too as there is no longer enough memory in the python process.

#test.py
import numpy as np
N = 20_000_000

def get_data():
    d = np.random.rand(N,100)
    return d

def process_data(d):
    return np.diff(d)

def test1():
    d = get_data()
    process_data(d)
    assert False  #simulate failure
    
def test2():
    d = get_data()
    process_data(d)

I used objgraph to trace the reference to the array. It looks like a frame used by the traceback hold onto it.

I think this could be a bug in pytest and may be there is a known work around for this issue.

Pytest 8.1.1 Python 3.11.0 OS: Ubuntu 22.04.3 LTS Install Packages Package Version iniconfig 2.0.0 numpy 1.26.4 packaging 24.0 pip 23.3.1 pluggy 1.4.0 psutil 5.9.8 pytest 8.1.1 setuptools 68.2.2 wheel 0.41.2

• [x] a detailed description of the bug or problem you are having
• [x] output of pip list from the virtual environment you are using
• [x] pytest and operating system versions
• [x] minimal example if possible

[RonnyPfannschmidt]

this is a unfortunate side-effect of ensuring staying debugable

there was no effort yet to provide a api/hook to purge tracebakcs/stacktraces of expensive resources

in this case im considering the array a "resource" as its something that puts possibly large pressure onto the system and ought to be purged if disposable

[ngchihuan]

Hi @RonnyPfannschmidt, Thanks a lot for the follow-up. Wondering what was the reason for not providing an api/hook to remove the traceback? Or the pytest team does not have this in plans at all? Is it complicated for a beginner to pytest repo like me to try to implement it?

[ngchihuan]

For others who also encounter this issue in their tests, I found a work-around

import weakref
import numpy as np
import pytest

N = 20_000_000

def get_data(n):
    d = np.random.rand(n,100)
    return d

def process_data(d):
    return np.diff(d)

def test1(data_provider):
    d = data_provider.get(N)
    process_data(d)
    assert False
    
def test2():
    d = get_data(N)
    process_data(d)
    
class DataFlyweight:
    def __init__(self):
        self._data =[]
        
    def get(self,n):
        if n not in self._data:
            self._data.append(get_data(n))
        return weakref.proxy(self._data[-1])

    def clear(self):
        self._data.clear()
    
@pytest.fixture
def data_provider():
    data_flyweight = DataFlyweight()
    yield data_flyweight
    data_flyweight.clear()

Memory does not explode and the second test passed as expected.

[RonnyPfannschmidt]

Multiple reasons

Initially tracebacks where not attached to exceptions and extracting ex info typically lost a number of locals automatically

Also for most test suites memory intensive variables where no issue

So there was no incentive/pressure to work on a feature with limited win and excessive possible edge cases for a long time