OpenTelemetry Support

[tharwan]

First check

• [X] I added a descriptive title to this issue.
• [X] I used the GitHub search to find a similar request and didn't find it.
• [X] I searched the Prefect documentation for this feature.

Prefect Version

2.x

Describe the current behavior

Prefect can be instrumented with OpenTelemetry manually, covering some parts of the execution.

Describe the proposed behavior

It would be great if prefect could be automatically instrumented via OpenTelemetry similar to many other python packages: https://opentelemetry-python-contrib.readthedocs.io/en/latest/

This would help to trace interactions of prefect flows with other parts of the data stack.

A first step could be that flows and tasks automatically generate spans which include attributes such as the name, the environment or parameters.

A second step might be that prefect Orion also adds spanns to a trace when a flow run is started from there.

Example Use

from opentelemetry.instrumentation.prefect import PrefectInstrumentor
from prefect import flow

@flow(instrumentor=PrefectInstrumentor)
def foo():
    pass

Additional context

No response

[billpalombi]

Thanks @tharwan. We definitely want Prefect flows to be observable - I've added this to our backlog.

[JimGorman17]

Thanks @tharwan. We definitely want Prefect flows to be observable - I've added this to our backlog.

@billpalombi Is there an active branch for this work?

[peteski22]

Thanks @tharwan. We definitely want Prefect flows to be observable - I've added this to our backlog.

@billpalombi Is there an active branch for this work?

I think this might be a bad sign:

cicdw removed observability status:accepted on Jul 19, 2024

Did it go from accepted, to ... thrown out? 😞

EDIT: I can't easily find a changelog that says OpenTelemetry was added (apologies if it exists) but I saw some closed issues that seem to indicate that OpenTelemetry is being configured out of the box: https://github.com/PrefectHQ/prefect/issues/17190 🤔

[chrisguidry]

Hi folks! Just wanted to give an update on this one. Prefect does instrument flow runs and task runs with OpenTelemetry spans and will propagate those traces across process boundaries. We added this as part of an experimental feature for Prefect Cloud that hasn't been released yet, but the bonus here is that this is just "regular" OpenTelemetry and these spans can be exported from your running flows to any OTel collector you like.

What I'm describing here is in Prefect 3.1.12 and above, and we don't have plans to backport this to Prefect 2 at this time.

By default, Prefect installs the opentelemetry-api package and adds spans around flow and task execution. These are richly annotated with events for state transitions and attributes describing the parameters/inputs. If a flow or task calls run_deployment, we propagate the current trace over to the new subflow as a linked span so you can trace the causal relationship.

This is all enabled by default and latent in the prefect library. You can disable the creation of spans by setting PREFECT_CLOUD_ENABLE_ORCHESTRATION_TELEMETRY=False, as some folks may not want these additional spans being produced.

The way the Python OpenTelemetry libraries work, there's a split between the instrumentation of code and exporting those spans off to another system for recording. Prefect doesn't enable any particular kind of exporting by default. For users of Prefect Cloud, if no other OpenTelemetry export has been enabled, these spans are shipped off to our OTel collector as part of an experimental feature. Otherwise, the spans will be shipped off to whatever exporter you have configured in the process (or nowhere, by default)

The quickest way to start exporting these spans to your own observability tools is with OpenTelemetry's "zero-code" approach through the opentelemetry-instrument CLI. This is part of the opentelemetry-sdk family of packages, and we include these by default in our prefecthq/prefect:3-* Docker images. If you're installing Prefect yourself, include the 'otel' extra (uv pip install prefect[otel]) to get this.

Once you have the opentelemetry-instrumentation* package, you'd invoke your entrypoint with the /usr/local/bin/opentelemetry-instrument wrapper, passing any arguments or envvars as appropriate to configure where your OTel collector is. When using a Prefect work pool, you'd give a custom command for your work pool that wrapped the prefect flow with /usr/local/bin/opentelemetry-instrument in the same way. For larger setups in Kubernetes, the OpenTelemetry Operator can save a lot of manual configuration by injecting some of this for you.

Thanks for being interested in OpenTelemetry support in Prefect. It's still fairly new, so your feedback is very welcome!

[cBournhonesque]

It looks like PREFECT_CLOUD_ENABLE_ORCHESTRATION_TELEMETRY=False prevents the creation of otel exporters (https://github.com/PrefectHQ/prefect/blob/3.2.7/src/prefect/telemetry/bootstrap.py#L18) but if the user has their own exporter, the prefect spans are still included.

I would have expected a setting that would allow me to completely get rid of prefect spans

[lghinet]

hello @chrisguidry , is there a way to enable otel instrumentation on self hosted kubernetes prefect workers ?

thanks