Disable pytest logging plugin for single test

[rooterkyberian]

What's the problem this feature will solve?

I'm trying to test a CLI tool which produces stdout/stderr output through logging. I need to make sure the logging is properly configured and stdout/stderr is indeed produced with proper formatting. I don't want to check configuration, but actual output of the command while retaining ability to mock things (this is why running cli tool in subprocess call is not an option).

Describe the solution you'd like

Any solution which won't easily break with pytest upgrade and prevent pytest from messing with logging configuration will be good. Ideally pytest would also be able to restore the logging configuration before test was started, but that seems like a another feature.

Solution ideas:

• disabled() for caplog - this would be consistent with https://docs.pytest.org/en/6.2.x/reference.html?highlight=disabled#pytest.CaptureFixture.disabled

• ability to unregister/register logging plugin - this is in theory possible but will break on registration in pytest_addoption as we try to "re-add" the same options all over again

def suspend_logcapture(pytestconfig):
    manager = pytestconfig.pluginmanager
    plugin = manager.get_plugin("logging-plugin")
    manager.unregister(plugin)
    yield
    manager.register(plugin)  # currently breaks on pytest_addoption

Alternative Solutions

right now I'm using following which I consider bad since I rely on pytest internals:

@pytest.fixture
def suspend_logcapture(pytestconfig):
    from _pytest.logging import LogCaptureHandler
    @contextlib.contextmanager
    def context(*args, **kwargs):
        yield LogCaptureHandler()

    with patch("_pytest.logging.catching_logs", context):
        yield

Tried also going through temporary removing handlers & settings set by pytest

@pytest.fixture
def suspend_logcapture():
    logger = logging.getLogger()
    config = {
        "disabled": False,
        "handlers": [],
        "level": logging.NOTSET,
    }
    old_config = {attr: getattr(logger, attr) for attr in config}
    for attr, value in config.items():
        setattr(logger, attr, value)
    yield
    for attr, value in old_config.items():
        setattr(logger, attr, value)

but pytest thinks nothing of it, as somehow the logging plugin is given yet another chance to modify root logger before the actual test gets run and sees its handler was removed and readds it

Additional context

solution ideas are based on: https://github.com/kvas-it/pytest-console-scripts https://pypi.org/project/pytest-disable-plugin/

As to why I didn't use them - code audits are a thing and costs time; also the core the problem seemed to me like more and issue with pytest itself.

[rooterkyberian]

for time being I'm going ahead with

@contextlib.contextmanager
def temporary_root_logger():
    logger = logging.getLogger()
    config = {
        "disabled": False,
        "handlers": [],
        "level": logging.NOTSET,
    }
    old_config = {attr: getattr(logger, attr) for attr in config}
    for attr, value in config.items():
        setattr(logger, attr, value)
    yield
    for attr, value in old_config.items():
        setattr(logger, attr, value)

Which is similar to previusly proposed yield fixture, with a difference that this is context manager which has to be explicitly called during tests (so not exactly pytest-way), and this time it works correctly (as opposed when doing the same thing as a yield fixture).

[matthewhughes934]

I'm not sure I understand your exact use-case, but what about using caplog.at_level?

import logging

import pytest

logger = logging.getLogger()


def logging_function():
    logger.debug("Here is some debug")
    logger.info("Here is some info")
    logger.warning("Here is some warning")
    logger.error("Here is some error")


@pytest.fixture
def silent_logging(caplog):
    # or maybe logging.CRITICAL + 1 if you want to remove
    # _everything_
    with caplog.at_level(logging.CRITICAL):
        yield


def test_with_logging(caplog, silent_logging):
    logging_function()

    assert not caplog.record_tuples

[rooterkyberian]

M use case is "I have a CLI program that I expect to properly configure Logger so everything logged is printed on stdout". But pytest messes with that Logger configuration and makes it impossible to empirically test if it was configured to print on stdout or not. Using caplog does not help because it verifies only if logger is used. What I want is to verify stdout was produced by logger.

[endrift]

~~I'm having a similar issue. I want to test the flow of the logging configuration in a program I'm working on, but pytest overrides it and takes over the logging flow so I can't actually test it.~~ Sorry for the false reply, there was actually a bug on my end.

[clemensvonschwerin]

Don't know if it is still relevant, but if you stumble across this thread as I did, the current version of pytest provides a cool builtin way of capturing logs to stdout and stderr separately by using the capsys fixture, a tutorial is available here: https://docs.pytest.org/en/6.2.x/capture.html

[svaningelgem]

Thanks for the tip @rooterkyberian . I went with this version in my conftest.py:

@pytest.fixture
def caplog(caplog: pytest.LogCaptureFixture) -> pytest.LogCaptureFixture:
    root = logging.getLogger()

    with unittest.mock.patch.object(root, 'disabled', new=False),\
        unittest.mock.patch.object(root, 'handlers', new=[]), \
        unittest.mock.patch.object(root, 'level', new=logging.NOTSET):
        yield caplog

Or those using the (excellent) pytest-mock library:

@pytest.fixture
def caplog(mocker: MockerFixture, caplog: pytest.LogCaptureFixture) -> pytest.LogCaptureFixture:
    root = logging.getLogger()

    mocker.patch.object(root, 'disabled', new=False)
    mocker.patch.object(root, 'handlers', new=[])
    mocker.patch.object(root, 'level', new=logging.NOTSET)

    return caplog

This just overwrites the default caplog fixture for your project. It'll allow you to capture all log output, and handle it as you deem necessary. However this log output isn't propagated to your client with the live log call output.

[istvans]

I think I had a very similar problem. I had a logging setup issue with one of our scripts. I wanted to write a simple pytest test that would fail with no logs at all with the original code and would pass with at least one log after the fix. There were always logs captured no matter what.

To give you a concrete example: even if you do not call logging.basicConfig and your script does not produce any logs, you will still see all the logs with caplog.