openai-agents doesn't seem to clear GLOBAL_TRACE_PROVIDER list

[codefromthecrypt]

Description

If I run this code with the correct ENV variables, it sends to logfire, but also has an error like this:

Tracing client error 401: {
  "error": {
    "message": "Incorrect API key provided: unused. You can find your API key at https://platform.openai.com/account/api-keys.",
    "type": "invalid_request_error",
    "param": null,
    "code": "invalid_api_key"
  }
}

code

import os

import httpx
import logfire
from agents import Agent, ModelSettings, OpenAIProvider, RunConfig, Runner, function_tool
from agents.tracing import GLOBAL_TRACE_PROVIDER

GLOBAL_TRACE_PROVIDER.shutdown()
GLOBAL_TRACE_PROVIDER.set_processors([])

logfire.configure()
logfire.instrument_openai_agents()


@function_tool
def get_latest_elasticsearch_version() -> str:
    """Returns the latest GA version of Elasticsearch in "X.Y.Z" format."""

    response = httpx.get("https://artifacts.elastic.co/releases/stack.json")
    response.raise_for_status()
    releases = response.json()["releases"]

    # Fetch releases and filter out non-release versions (e.g., -rc1) and
    # remove any " GA" suffix.
    versions = []
    for r in releases:
        v = r["version"].removesuffix(" GA")
        if "-" in r["version"]:
            continue
        versions.append(v)

    if not versions:
        raise ValueError("No valid versions found")

    # "8.9.1" > "8.10.0", so coerce to a numeric tuple: (8,9,1) < (8,10,0)
    return max(versions, key=lambda v: tuple(map(int, v.split("."))))


def main():
    model_name = os.getenv("CHAT_MODEL", "gpt-4o-mini")
    model = OpenAIProvider(use_responses=False).get_model(model_name)
    agent = Agent(
        name="version_assistant",
        tools=[get_latest_elasticsearch_version],
        model=model,
        model_settings=ModelSettings(temperature=0),
    )

    result = Runner.run_sync(
        starting_agent=agent,
        input="What is the latest version of Elasticsearch?",
        run_config=RunConfig(workflow_name="GetLatestElasticsearchVersion"),
    )
    print(result.final_output)


if __name__ == "__main__":
    main()

If I paste this above logfire.configure(), it is ok

GLOBAL_TRACE_PROVIDER.shutdown()
GLOBAL_TRACE_PROVIDER.set_processors([])

Python, Logfire & OS Versions, related packages (not required)

logfire="3.8.0"
platform="macOS-15.3.1-arm64-arm-64bit"
python="3.12.8 | packaged by conda-forge | (main, Dec  5 2024, 14:19:53) [Clang 18.1.8 ]"
[related_packages]
requests="2.32.3"
pydantic="2.10.6"
openai="1.66.2"
protobuf="5.29.3"
rich="13.9.4"
executing="2.2.0"
opentelemetry-api="1.30.0"
opentelemetry-exporter-otlp="1.30.0"
opentelemetry-exporter-otlp-proto-common="1.30.0"
opentelemetry-exporter-otlp-proto-grpc="1.30.0"
opentelemetry-exporter-otlp-proto-http="1.30.0"
opentelemetry-instrumentation="0.51b0"
opentelemetry-instrumentation-system-metrics="0.51b0"
opentelemetry-proto="1.30.0"
opentelemetry-resource-detector-azure="0.1.5"
opentelemetry-resourcedetector-gcp="1.8.0a0"
opentelemetry-sdk="1.30.0"
opentelemetry-sdk-extension-aws="2.1.0"
opentelemetry-semantic-conventions="0.51b0"

[alexmojaki]

I don't think we should clear the processors by default. People will probably want to see the logs in the official OpenAI UI. But I agree it would be nice to have a convenient way to turn that off, like a kwarg in instrument_openai_agents. Would you be willing to open a PR?

[codefromthecrypt]

ok I looked into it. I misunderstood the implementation at first. Looks like we are wrapping. I think it is a coin toss what is intuitive. Personally, I think it is more intuitive to replace the tracer than append to it, but I understand that is subjective. Below is assuming we want to go down the path to double-trace by default.

So, the main thing is that if you are using ollama, grok, any number of LLM providers emulating OpenAI, they don't have the "/traces/ingest" endpoint, so this is why you get a 401. Even if you were using OpenAI, you may want to avoid the extra overhead if you are only using logfire. I wrote a doc around this and if you agree, proceed later to the implications I wrote below it.

    def instrument_openai_agents(self, wrap_global_tracer: bool = True) -> None:
        """Instrument the [`agents`](https://github.com/openai/openai-agents-python) framework from OpenAI.

        For instrumentation of the OpenAI Agents SDK package,
        see [`instrument_openai()`][logfire.Logfire.instrument_openai].

        Args:
            wrap_global_tracer: If True, calls the existing OpenAI Agents SDK global tracer before calling the one
                in Logfire. By default, the OpenAI Agents SDK configures a tracer that exports to the "/traces/ingest"
                endpoint. Setting this to False avoids exporting also to this endpoint, which may not be available when
                using LLM providers that emulate OpenAI's API. False also reduces trace overhead for the same reason.
        """

If we do this, I think we should decouple our TracerProvider from the wrapper, so that it can be called directly. This would avoid overhead and also result in smaller stacks when logfire is enabled. However, at the moment the code we want is conflated with wrapping the global tracer. That's why we should agree on the path forward first.

Meanwhile, here's my workaround

# Shut down the global tracer as it sends to the OpenAI "/traces/ingest"
# endpoint, which we aren't using and doesn't exist on alternative backends
# like Ollama.
GLOBAL_TRACE_PROVIDER.shutdown()

logfire.configure()
logfire.instrument_openai_agents()

[alexmojaki]

I think we will eventually switch to using a trace processor. I'd rather not mention implementation details in the docs or argument names.

I also don't think it's worth worrying about the overhead of OpenAI's tracing, whether it's enabled or not.

[codefromthecrypt]

If we move to a trace processor. Wouldn't it break the ability to propagate context (since it is like a batch exporter?) Meaning you couldn't combine this with other logfire instrumentation if you did? Maybe I am missing something...

[alexmojaki]

There's some small nuances around context propagation, but no, the processor would be synchronous and delegate to OTel processors (including BatchSpanProcessor) so it would know the current span.

[codefromthecrypt]

ok fair enough, just the type SynchronousMultiTracingProcessor scared me about there being a non-synchronous future which would likely break things. My 2p is add tests to make sure you know if that occurs as we don't want the http or otherwise children to become new traces or attached to the wrong one.

[alexmojaki]

I was thinking logfire.instrument_openai_agents(disable_other_tracing=True) (maybe with a more concise name) would clear other processors, both now and in a future where we use a processor ourselves.

[codefromthecrypt]

replace_processors=True? I think whether we replace the tracer or not we are replacing the processors. If we say we are disabling other tracing, we should do that. As a benchmark guy, I know I'm nitpicking, but leaving around infra that isn't used is cringe. Let's say what we are willing to do, and not more?

[alexmojaki]

replace_processors doesn't sound like it's turning off attempts to send to OpenAI.

[codefromthecrypt]

gotcha, anyway disable_other_tracing gives us flexibility to straight-pipe to logfire should we decide to later

[codefromthecrypt]

ps sorry I am slow on this. I have a conference this week which has unrelated work: allthingsopen.ai, so heads down on that. Someone can yank this if they like

[dast-draftwise]

Any update on this?

[codefromthecrypt]

sorry I've switched jobs and this particular piece isn't in my foreground at the moment. Please no-one block on me