🚀 Feature: Make instrumentations compatible with "opentelemetry-instrument" automatic instrumentation

[torbenw]

Which component is this feature for?

All Packages

🔖 Feature description

When starting a python app via "opentelemetry-instrument python myapp.py" available instrumentors are automatically discovered and used for tracing. This works with official opentelemetry instrumentors, but not with the instrumentors created by traceloop. Instrumentors created by traceloop should behave the same way as the opentelemetry instrumentors so that they can be used in automatic instrumentation without having to change the code of the python app which is being observed.

🎤 Why is this feature needed ?

Being able to use automatic instrumentation would remove the requirement of changing code to add observability to apps.

✌️ How do you aim to achieve this?

The "official" instrumentations by opentelemetry have the following lines in their pyproject.toml:

[project.entry-points.opentelemetry_instrumentor]
httpx = "opentelemetry.instrumentation.httpx:HTTPXClientInstrumentor"

Instrumentations with the "opentelemetry_instrumentor" entry-point are automatically loaded by the auto-instrumentation:

for entry_point in iter_entry_points("opentelemetry_instrumentor"):
  ...
  distro.load_instrumentor(entry_point, skip_dep_check=True)

🔄️ Additional Information

No response

👀 Have you spent some time to check if this feature request has been raised before?

• [X] I checked and didn't find similar issue

Are you willing to submit PR?

None

[nirga]

Thanks @torbenw seems like an easy one liner fix! We'll definitely do that, just need to make sure it's indeed working.

[npuichigo]

@nirga thanks for this update. Any eta to publish a new version so we can use it for auto instrumentation with opentelemetry-instrument?

[nirga]

@torbenw I’m publishing now

[torbenw]

This does not seem to work for me. Maybe there's more that has to be changed? I'm new to Python, so I don't really know how things work with installing packages.

For "opentelemetry-instrumentation-httpx" I have a file "entry_points.txt" in .../venv/Lib/site-packages/opentelemetry_instrumentation_httpx-0.44b0.dist-info:

[opentelemetry_instrumentor]
httpx = opentelemetry.instrumentation.httpx:HTTPXClientInstrumentor

In .../venv/Lib/site-packages/opentelemetry_instrumentation_langchain-0.14.1.dist-info there is no such file.

[nirga]

@torbenw I'll try to ask around in the community in circle back.

[nirga]

@torbenw great, seems that poetry doesn't support PEP-621 :/ Looks like they've been discussing it there for a while, and there's a recent open PR that tries to solve it. I'll try to help and make it get merged there so it will be resolved for us as well. But until then, there isn't an easy solution for us here unfortunately :/

[torbenw]

I think there's more to do even if we can get the instrumentations into the entry_points.txt. I've manually added those files to the langchain and openai instrumentation to check if it would work if they exist, but those instrumentations aren't used in the traces and there are errors:

Instrumenting of langchain failed
Traceback (most recent call last):
  File "...\venv\lib\site-packages\pkg_resources\__init__.py", line 2827, in requires
    deps.extend(dm[safe_extra(ext)])
KeyError: 'instruments'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "...\venv\lib\site-packages\opentelemetry\instrumentation\auto_instrumentation\_load.py", line 77, in _load_instrumentors
    conflict = get_dist_dependency_conflicts(entry_point.dist)
  File "...\venv\lib\site-packages\opentelemetry\instrumentation\dependencies.py", line 32, in get_dist_dependency_conflicts
    for dep in dist.requires(("instruments",)):
  File "...\venv\lib\site-packages\pkg_resources\__init__.py", line 2829, in requires
    raise UnknownExtra(
pkg_resources.UnknownExtra: opentelemetry-instrumentation-langchain 0.14.1 has no such extra feature 'instruments'
Failed to auto initialize opentelemetry
Traceback (most recent call last):
  File "...\venv\lib\site-packages\pkg_resources\__init__.py", line 2827, in requires
    deps.extend(dm[safe_extra(ext)])
KeyError: 'instruments'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "...\venv\lib\site-packages\opentelemetry\instrumentation\auto_instrumentation\sitecustomize.py", line 39, in initialize
    _load_instrumentors(distro)
  File "...\venv\lib\site-packages\opentelemetry\instrumentation\auto_instrumentation\_load.py", line 91, in _load_instrumentors
    raise exc
  File "...\venv\lib\site-packages\opentelemetry\instrumentation\auto_instrumentation\_load.py", line 77, in _load_instrumentors
    conflict = get_dist_dependency_conflicts(entry_point.dist)
  File "...\venv\lib\site-packages\opentelemetry\instrumentation\dependencies.py", line 32, in get_dist_dependency_conflicts
    for dep in dist.requires(("instruments",)):
  File "...\venv\lib\site-packages\pkg_resources\__init__.py", line 2829, in requires
    raise UnknownExtra(
pkg_resources.UnknownExtra: opentelemetry-instrumentation-langchain 0.14.1 has no such extra feature 'instruments'

[nirga]

@torbenw Hmm, that still seems to be related to the entry point. Looks like it can't find the method that instruments (although its there).

[torbenw]

I think this is missing, too?

Example:

[project.optional-dependencies]
instruments = [
  "fastapi ~= 0.58",
]

pkg_resources.UnknownExtra: opentelemetry-instrumentation-langchain 0.14.1 has no such extra feature 'instruments'

I might be misunderstanding this, I'm a Java developer with almost no Python experience. ;)

[nirga]

Hmm you're right I see that in other instrumentations. But it still won't work since we use Poetry and this attribute won't get bundled (if I understood correctly). So until Poetry supports PEP-621, or we switch to another build system (which will have its own issues since we have a monorepo here) we won't be able to support it :/

[nirga]

I think there's a solution! 🤔

[npuichigo]

how does it works?

[nirga]

@npuichigo just install any of the instrumentations and then initialize the auto instrumentation:

pip install opentelemetry-instrumentation-openai
pip install opentelemetry-distro opentelemetry-exporter-otlp
opentelemetry-bootstrap -a install

opentelemetry-instrument python myapp.py

[kguelzau]

Has anybody used this in a custom image used for the open telemetry operator? The default image used for this mechanism is here. I wonder what needs to be done to also use the openllmetrym libs.

My naive test to just add this to requirements.txt did not work (error.txt):

opentelemetry-instrumentation-anthropic==0.23.0
opentelemetry-instrumentation-bedrock==0.23.0
opentelemetry-instrumentation-chromadb==0.23.0
opentelemetry-instrumentation-cohere==0.23.0
opentelemetry-instrumentation-haystack==0.23.0
opentelemetry-instrumentation-langchain==0.23.0
opentelemetry-instrumentation-llamaindex==0.23.0
opentelemetry-instrumentation-openai==0.23.0
opentelemetry-instrumentation-pinecone==0.23.0
opentelemetry-instrumentation-qdrant==0.23.0
opentelemetry-instrumentation-replicate==0.23.0
opentelemetry-instrumentation-transformers==0.23.0
opentelemetry-instrumentation-vertexai==0.23.0
opentelemetry-instrumentation-watsonx==0.23.0
opentelemetry-instrumentation-weaviate==0.23.0

[nirga]

Hey @kguelzau sorry, this slipped between the cracks. I'll look into this!

[kguelzau]

@nirga perhaps there is still a problem in the build process. The other instrumentations include something like this in the resulting METADATA file: Requires-Dist: aiohttp~=3.0; extra == 'instruments' This is still missing for the openllmetry instrumentations.

[kguelzau]

@nirga / @galkleinman I am also not a python dev, but I think you need to change the dependency to an optional dependency like in the draft PR #2191 (see https://python-poetry.org/docs/pyproject/#extras). Doing a poetry build results in a

Provides-Extra: instruments
Requires-Dist: openai[datalib] (>=1.50.0) ; extra == "instruments" 

So this should hopefully work within a custom image for the open telemetry operator.

[nirga]

@kguelzau weird - the package is in the dev dependencies so it shouldn't appear in the resulted pyproject. Have you tried building it with the previous config and see what happens?

[feng-95]

It seems the entry-points doesn't works well as expected. I have installed the opentelemetry-instrumentation-openai but didn't use openai in my project, but the openai-instrumentor still stared, and then report an error:

Importing of openai failed, skipping it
Traceback (most recent call last):
  File "/otel-auto-instrumentation-python/opentelemetry/instrumentation/auto_instrumentation/_load.py", line 116, in _load_instrumentors
    distro.load_instrumentor(entry_point, skip_dep_check=True)
  File "/otel-auto-instrumentation-python/opentelemetry/instrumentation/distro.py", line 61, in load_instrumentor
    instrumentor: BaseInstrumentor = entry_point.load()
  File "/otel-auto-instrumentation-python/importlib_metadata/__init__.py", line 189, in load
    module = import_module(match.group('module'))
  File "/usr/local/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 843, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/otel-auto-instrumentation-python/opentelemetry/instrumentation/openai/__init__.py", line 6, in <module>
    from opentelemetry.instrumentation.openai.shared.config import Config
  File "/otel-auto-instrumentation-python/opentelemetry/instrumentation/openai/shared/__init__.py", line 2, in <module>
    import openai
ModuleNotFoundError: No module named 'openai'

When I changed the opentelemetry-instrumentation-openai project.yaml to hatchling building system, this problem has been fixed.

[feng-95]

Hi @nirga Could you please take a look to this again? Will you try to fix this incompatible with entry-points? Or you need to wait until Poetry supports PEP-621?