opentelemetry-python
opentelemetry-python copied to clipboard
open-telemetry
Traces are not exported to Jaeger
I've been trying to run the basic Open Telemetry "Getting Started" guide, but ran into multiple issues. I tried running it on a RHEL8 machine (tried fedora and MACOS as well) while using pipenv, and python 3.9.6. I ran the latest version of the library. I have the following packages installed on the environment:
anyio==3.6.1
asgiref==3.5.2
backoff==2.1.2
certifi==2022.6.15
charset-normalizer==2.1.0
click==8.1.3
Deprecated==1.2.13
fastapi==0.79.0
Flask==2.1.3
googleapis-common-protos==1.56.2
grpcio==1.48.0rc1
h11==0.13.0
httptools==0.4.0
idna==3.3
importlib-metadata==4.12.0
itsdangerous==2.1.2
Jinja2==3.1.2
MarkupSafe==2.1.1
opentelemetry-api==1.12.0rc2
opentelemetry-distro==0.32b0
opentelemetry-exporter-jaeger==1.12.0rc2
opentelemetry-exporter-jaeger-proto-grpc==1.12.0rc2
opentelemetry-exporter-jaeger-thrift==1.12.0rc2
opentelemetry-exporter-otlp-proto-grpc==1.12.0rc2
opentelemetry-instrumentation==0.32b0
opentelemetry-instrumentation-asgi==0.32b0
opentelemetry-instrumentation-fastapi==0.32b0
opentelemetry-instrumentation-requests==0.32b0
opentelemetry-proto==1.12.0rc2
opentelemetry-sdk==1.12.0rc2
opentelemetry-semantic-conventions==0.32b0
opentelemetry-util-http==0.32b0
protobuf==3.20.1
pydantic==1.9.1
python-dotenv==0.20.0
PyYAML==6.0
requests==2.28.1
six==1.16.0
sniffio==1.2.0
starlette==0.19.1
thrift==0.16.0
typing_extensions==4.3.0
urllib3==1.26.11
uvicorn==0.18.2
uvloop==0.16.0
watchfiles==0.16.0
websockets==10.3
Werkzeug==2.2.0
wrapt==1.14.1
zipp==3.8.1
Steps to reproduce
Same exact example as in the getting started guide (including all the dependencies and the bootstrap). I'm running the flask app (which is working fine) with the automated telemetry, and exporting the traces to a Jaeger instance (using docker and the all-in-one command presented).
only difference is, the docs refer to this import: from opentelemetry.exporter.jaeger import JaegerExporter which does not exist and can't be imported. instead, I'm using thrift (tried grpc as well, encountered the same issues).
code example for setting up the exporter:
from opentelemetry.exporter.jaeger.thrift import JaegerExporter
from opentelemetry.sdk.resources import SERVICE_NAME, Resource
from collections import Counter
from typing import Mapping
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from flask import Flask, request
app = Flask(__name__)
counter = Counter()
# thrift:
resource = Resource(attributes={
SERVICE_NAME: "counter"
})
jaeger_exporter = JaegerExporter(
agent_host_name="localhost",
agent_port=6831,
)
provider = TracerProvider(resource=resource)
processor = BatchSpanProcessor(jaeger_exporter)
provider.add_span_processor(processor)
trace.set_tracer_provider(provider)
command that I use:
opentelemetry-instrument --traces_exporter jaeger --metrics_exporter console --logs_exporter console flask run
What is the expected behavior? The traces in the Jaeger instance, under a service called "counter".
What is the actual behavior?
While the traces can be exported to console (when set to export to console), there is no trace of them in the Jaeger instance, there is only jaeger-query service available. Instead, I'm getting the following error:
Failed to export batch. Status code: StatusCode.UNAVAILABLE
E0725 17:39:22.176535710 320299 ssl_transport_security.cc:1495] Handshake failed with fatal error SSL_ERROR_SSL: error:100000f7:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER.
as well as:
Overriding of current TracerProvider is not allowed
This does not happen when exporting to the console.
only difference is, the docs refer to this import:
from opentelemetry.exporter.jaeger import JaegerExporterwhich does not exist and can't be imported.
only difference is, the docs refer to this import:
from opentelemetry.exporter.jaeger import JaegerExporterwhich does not exist and can't be imported.+1 (problem docs source)
I see now that there's already a PR to solve this issue. :)
You can solve this without waiting for the pr. Jaeger Thrift exporter is not using port 6831 rather 5775 and sending over udp. Therefore in your local docker-compose open port 5775 for udp: ` jaeger: image: jaegertracing/all-in-one restart: always ports: - 16686:16686 - 6831:6831 - 5775:5775/udp
` Then in your code use port 5775.
That did he trick for me
@odedlevy02 Unfortunately didn't work for me. also, the Jaeger docs refer to 5775 as (deprecated) accept zipkin.thrift over compact Thrift protocol (used by legacy clients only). any chance you're using it with zipkin?
Not that I know of. This is what I am using:
from opentelemetry.exporter.jaeger.thrift import JaegerExporter
And it worked for me
On Thu, Jul 28, 2022 at 5:38 PM Omer Amsalem @.***> wrote:
@odedlevy02 https://github.com/odedlevy02 Unfortunately didn't work for me. also, the Jaeger docs refer to 5775 as (deprecated) accept zipkin.thrift over compact Thrift protocol (used by legacy clients only). any chance you're using it with zipkin?
— Reply to this email directly, view it on GitHub https://github.com/open-telemetry/opentelemetry-python/issues/2842#issuecomment-1198231518, or unsubscribe https://github.com/notifications/unsubscribe-auth/AD6KFB75L3RYAWBOHLUBIS3VWKLNJANCNFSM54VXVX5A . You are receiving this because you were mentioned.Message ID: @.***>
I'm not sure what the issue was, but using otlp instead of jaeger as the traces_exporter solved all issues and errors.
Overriding of current TracerProvider is not allowed
There can only be one global tracer provider and opentelemetry-instrument already set one so if you try to set again in your code it has no effect and shows the warning.
Failed to export batch. Status code: StatusCode.UNAVAILABLE E0725 17:39:22.176535710 320299 ssl_transport_security.cc:1495] Handshake failed with fatal error SSL_ERROR_SSL: error:100000f7:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER.
Continuing the above point, If you are already trying to set the exporter using opentelemetry-instrument your exporter in code will not be added to global tracer pipeline set by auto instrumentation. And the --traces_exporter jaeger configures the jaeger gRPC exporter which uses secure connection unless explicitly set insecure either vai param or env https://opentelemetry-python.readthedocs.io/en/latest/sdk/environment_variables.html#opentelemetry.sdk.environment_variables.OTEL_EXPORTER_JAEGER_GRPC_INSECURE. If you want to add thrift exporter you should use --traces_exporter jaeger_thrift.
Is this still an issue? Please use the insecure init arg or the newly added env for insecure jaeger collector deployments. Feel free to re-open if you still run into this problem.
@srikanthccv Hello. You say "so if you try to set again in your code it has no effect and shows the warning."
This is clear from the error message, however you give no information on how to properly output to Jaeger.
I am reading this guide for python, all methods there use trace.set_tracer_provider(provider) and thus produce this error.
Could you please clarify what is correct workaround?
How to either set jaeger exporter for global provider if it's already initialized?
Or, how to override the global provider with newly created one, if that's even possible?
I mention in the second part of the comment how to pass the exporter as an argument and use env if you have to change things like endpoint etc... And there is no way to override the global provider.
@srikanthccv So to sum it up, correct approach to exporting to Jaeger is not importing opentelemetry modules into code, but rather just installing dependencies from pip and then using certain cmdline args for opentelemetry launching?
Is opentelemetry-exporter-jaeger package enough for --traces_exporter jaeger?
I assume you are using the auto instrumentation. When you use it with auto instrumentation, yes, that's the expected way to set the exporter. Since it already sets the global provider you would pass the {traces,metrics,logs} exporters to configure with comma separated list.
If you are also creating a trace provider in code and configuring with exporter it still works but that won't be global provider instrumentation packages used and one won't be able to emit any telemetry with it. But any tracer instances created with this provider will continue to send the data. Hope that clears up things.