Add `logfire.instrument_pytest()`

[intellectronica]

Description

I'd like to trace everything happening in my testsuite, with traces grouped by a span for each test case, so that I can observe the how my code behaves when called from the testsuite, assure myself that the tests are working correctly when they pass, or understand what isn't working if they fail.

See also: https://x.com/intellectronica/status/1815100560712102032

[Kludex]

@RonnyPfannschmidt Pinging you since it may be interesting to watch (or help, or discuss).

[RonnyPfannschmidt]

pytest instrumentation shouldn't be a monkeypatch, but rather a pytest plugin that provides hook wrappers against the key hooks of collecton and test session running

i beleive this should have 2 levels

1. pluggy instrumentation

implemnted via its trace functionality and/or a new hook point provided specifically for telemetry, providing deep details on all hook invocatons

2. pytest reporting via a plugin

using the hookwrapper functionality available for plugins, most of pytest can be wrapped in spans

the collection, test protocol and reporting hooks have to be targeted

id like to discuss alternate approaches of integrating this as well as im under the impression that different people/setups need different detail level and its not clear to me how filtering of span tress would work on a consumer/display side

my understanding of telemetry an d how to use it is still lacking and thats a impediment for designing a good integration

[jackmpcollins]

Here's some code I have added to my conftest.py to create a span for each test. However there is no indication of which tests have failed in the logfire UI. Having failed tests show as an exception in logfire would be helpful for quickly finding them - seems like this could be achieved using hookwrapper mentioned above but I have not looked into this.

# conftest.py

import logfire
import pytest

logfire.configure(send_to_logfire=True, service_name="tests")
logfire.instrument_openai()
logfire.instrument_anthropic()


@pytest.fixture(autouse=True)
def _trace_test(request):
    with logfire.span(request.node.name):
        yield

[RonnyPfannschmidt]

I recommend try first Hook wrappers for the hooks of the runtest protocol family

[seidtgeist]

Much over-engineered:

import logfire
import pytest

org_name = "your-org"
project_name = "your-project"
service_name = "your-service"

logfire.configure(
    console=False,
    show_summary=False,
    send_to_logfire=True,
    service_name=service_name,
)


def pytest_collection(session: pytest.Session):
    session.stash["session_span"] = logfire.span("test session")


def pytest_itemcollected(item: pytest.Item):
    if "parent_span" not in item.parent.stash:
        item.parent.stash["parent_span"] = logfire.span(item.parent.name)

    item.stash["item_span"] = logfire.span(item.name)


@pytest.hookimpl(wrapper=True)
def pytest_runtest_protocol(item: pytest.Item):
    with item.session.stash["session_span"]:
        with item.parent.stash["parent_span"]:
            with item.stash["item_span"]:
                yield


def pytest_exception_interact(node: pytest.Item, call: pytest.CallInfo):
    logfire.exception(str(call.excinfo.value), _exc_info=call.excinfo.value)

[RonnyPfannschmidt]

The low engineer version is completely misleading with various details and doesn't reflect fixture setup/teardown as well as test protocol stages in a meaningful way

[seidtgeist]

wasn't meant as criticism, your comment actually got me started on the path to writing the above snippet from scratch all morning 😇 has someone more talented made a proper hook-based plugin already?

[Kludex]

Not sure if it's interesting but I found this: https://www.cncf.io/blog/2024/11/04/opentelemetry-is-expanding-into-ci-cd-observability/