pytest-memray

pytest-memray copied to clipboard

Feature Request

Pytest can discover and run unittests but pytest-memray does not work on these. An example:

https://github.com/ChaoticRoman/pytest-monitor-example/tree/memray-example

There is an exactly same issue with alternative pytest memory profiler called pytest-monitor:

https://github.com/CFMTech/pytest-monitor/issues/39

I made very simple memory profiler for unittests that can be helpful for someone encountering this issue:

https://github.com/ChaoticRoman/pytest-monitor-example/blob/custom_memory_profiler/unittestit.py

Pytest can discover and run unittests but pytest-memray does not work on these.

Can you clarify what's the error here?

Pytest can discover and run unittests but pytest-memray does not work on these.

Can you clarify what's the error here?

There are two most common ways how to write tests in Python:

1. Using official unittest module: subclassing unittest.TestCase, tests are then those its methods with name starting with test_ prefix.
2. Using pytest extension module. Tests are all functions in top-level namespace with name starting with test_ prefix.

This example illustrates both these ways: https://github.com/ChaoticRoman/pytest-monitor-example/blob/memray-example/tests/test_float.py

The officially supported way is the one using unittest module and most open source and enterprise projects use this one. But pytest is very often used as a test discovery and test runner for these, because pytest does support both of these, it has more customizable output and there are many extensions e.g. for parallel execution, CI/CD integration and others. The unfortunate fact is that both available memory profiling extensions (pytest-monitor and pytest-memray) do not support other tests than those in native pytest format.

This 40 lines long script does implement memory profiler with unittests discovery:

https://github.com/ChaoticRoman/pytest-monitor-example/blob/custom_memory_profiler/unittestit.py

but it would be nice to have full support through a pytest extension, either via pytest-monitor or pytest-memray instead of three tools with incomplete features.

My question was why does it not support it? Doesn't just works? What happens if you try to use it?

@gaborbernat It does not :( Memory allocation is reported for native pytests but not for unittests (although those are discovered and executed by pytest). Using the example referenced above:

# pytest -v --memray tests/
========================================================================================== test session starts ===========================================================================================
platform linux -- Python 3.9.2, pytest-7.1.2, pluggy-0.13.0 -- /usr/bin/python3
cachedir: .pytest_cache
rootdir: /opt/pytest-monitor-example
plugins: memray-1.1.0, monitor-1.6.3
collected 6 items                                                                                                                                                                                        

tests/test_float.py::TestStringMethods::test_correct_string PASSED                                                                                                                                 [ 16%]
tests/test_float.py::TestStringMethods::test_empty_string PASSED                                                                                                                                   [ 33%]
tests/test_float.py::TestStringMethods::test_invalid_string PASSED                                                                                                                                 [ 50%]
tests/test_float.py::test_empty_string PASSED                                                                                                                                                      [ 66%]
tests/test_float.py::test_allocate_1MB PASSED                                                                                                                                                      [ 83%]
tests/test_float.py::test_allocate_100MB PASSED                                                                                                                                                    [100%]


============================================================================================= MEMRAY REPORT ==============================================================================================
Allocations results for tests/test_float.py::test_allocate_100MB

         📦 Total memory allocated: 107.5MiB
         📏 Total allocations: 200
         📊 Histogram of allocation sizes: |█|
         🥇 Biggest allocating functions:
                - <listcomp>:/opt/pytest-monitor-example/tests/test_float.py:34 -> 107.5MiB


Allocations results for tests/test_float.py::test_allocate_1MB

         📦 Total memory allocated: 1.1MiB
         📏 Total allocations: 122
         📊 Histogram of allocation sizes: |█|
         🥇 Biggest allocating functions:
                - <listcomp>:/opt/pytest-monitor-example/tests/test_float.py:34 -> 1.1MiB


=========================================================================================== 6 passed in 1.02s ============================================================================================

Just +1 on this, i've tried this today with a package using unittest.TestCase for all the tests and I don't get any memray output:

$ pytest -v --memray
====================================== test session starts =======================================
platform linux -- Python 3.8.13, pytest-7.1.2, pluggy-1.0.0 -- /bin/python
cachedir: .pytest_cache
rootdir: XXXXXX
plugins: memray-1.2.0
collected 21 items                                                                               

tests/test_agent.py::testAgentClass::test_get_journey PASSED                               [  4%]
tests/test_agent.py::testAgentClass::test_get_name PASSED                                  [  9%]
tests/test_agent.py::testAgentClass::test_get_name_unique PASSED                           [ 14%]
tests/test_agent.py::testAgentClass::test_get_position PASSED                              [ 19%]
tests/test_agent.py::testAgentClass::test_update PASSED                                    [ 23%]
tests/test_agent.py::testRandomWalkClass::test_instantiation PASSED                        [ 28%]
tests/test_agent.py::testRandomWalkClass::test_step PASSED                                 [ 33%]
tests/test_agent.py::testRandomWalkClass::test_update PASSED                               [ 38%]
tests/test_world.py::testWorldClass::test_get_name PASSED                                  [ 42%]
tests/test_world.py::testWorldClass::test_get_name_unique PASSED                           [ 47%]
tests/test_world.py::testWorldClass::test_get_population PASSED                            [ 52%]
tests/test_world.py::testWorldClass::test_limits PASSED                                    [ 57%]
tests/test_world.py::testWorldClass::test_populate PASSED                                  [ 61%]
tests/test_world.py::TestPopulationClass::test_instanstiation PASSED                       [ 66%]
tests/test_world.py::TestPopulationClass::test_spawn PASSED                                [ 71%]
tests/test_world.py::TestPopulationClass::test_spawn_fail PASSED                           [ 76%]
tests/test_world.py::TestToeGuardClass::test_instanstiation PASSED                         [ 80%]
tests/test_world.py::TestToeGuardClass::test_step PASSED                                   [ 85%]
tests/test_world.py::TestToeGuardClass::test_step_limits PASSED                            [ 90%]
tests/test_world.py::TestToeGuardClass::test_step_noShared PASSED                          [ 95%]
tests/test_world.py::TestToeGuardClass::test_step_value PASSED                             [100%]


========================================= MEMRAY REPORT ==========================================
======================================= 21 passed in 4.25s =======================================

Although in all fairness the pytest docs on using unittest says:

The following pytest features do not work, and probably never will due to different design philosophies:

• Fixtures (except for autouse fixtures, see below);
• Parametrization;
• Custom hooks;

Third party plugins may or may not work well, depending on the plugin and the test suite.

I'm unclear what changes might be required to make this work with unittests but will do some reading.

We use custom hooks, so unfortunately is very unlikely that we will be able to support it if pytest doesn't :(

Closing this as purest doesn't allow us to do this easily