openllmetry
openllmetry copied to clipboard
traceloop
🐛 Bug Report: traceloop-sdk does not work with Instana and/or OpenTelemetry
Which component is this bug for?
Traceloop SDK
📜 Description
When using traceloop-sdk with Instana and/or OpenTelemetry, the integration fails due to an AttributeError. The error appears when starting new spans and ending them with the BatchSpanProcessor from instana/opentelemetry-sdk. The relevant error message is:
AttributeError: 'BatchSpanProcessor' object has no attribute 'record_span'
This prevents spans from being properly recorded and exported, making tracing with Instana not possible using the provided setup.
👟 Reproduction steps
- Install the following packages:
- traceloop-sdk==0.47.3
- instana==3.9.0
- opentelemetry-api==1.38.0
- opentelemetry-sdk==1.38.0
- opentelemetry-exporter-otlp-proto-http==1.38.0
- Use the script below to initialize tracing and send traces to Instana (see 'Script used' section).
- Run the script.
- Observe the error in the logs:
AttributeError: 'BatchSpanProcessor' object has no attribute 'record_span'
👍 Expected behavior
Spans should be properly started and ended, and tracing data should be exported to Instana without error. Instana and OpenTelemetry should seamlessly work together with traceloop-sdk.
👎 Actual Behavior with Screenshots
The following error is thrown when attempting to end a span:
AttributeError: 'BatchSpanProcessor' object has no attribute 'record_span'
Full traceback:
WARNING:opentelemetry.trace:Overriding of current TracerProvider is not allowed
INFO:__main__:Auth operation completed
Traceback (most recent call last):
File "/tmp/.venv/lib/python3.12/site-packages/opentelemetry/trace/__init__.py", line 589, in use_span
yield span
File "/tmp/.venv/lib/python3.12/site-packages/instana/tracer.py", line 175, in start_as_current_span
yield span
File "/tmp/./traceloop_test.py", line 73, in main
with tracer.start_as_current_span("auth_operation") as auth_span:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/contextlib.py", line 144, in __exit__
next(self.gen)
File "/tmp/.venv/lib/python3.12/site-packages/instana/tracer.py", line 169, in start_as_current_span
with use_span(
^^^^^^^^^
File "/usr/local/lib/python3.12/contextlib.py", line 144, in __exit__
next(self.gen)
File "/tmp/.venv/lib/python3.12/site-packages/opentelemetry/trace/__init__.py", line 619, in use_span
span.end()
File "/tmp/.venv/lib/python3.12/site-packages/instana/span/span.py", line 200, in end
self._span_processor.record_span(self._readable_span())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'BatchSpanProcessor' object has no attribute 'record_span'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/tmp/./traceloop_test.py", line 93, in <module>
main()
File "/tmp/./traceloop_test.py", line 68, in main
with tracer.start_as_current_span("test_banking_workflow") as span:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/contextlib.py", line 158, in __exit__
self.gen.throw(value)
File "/tmp/.venv/lib/python3.12/site-packages/instana/tracer.py", line 169, in start_as_current_span
with use_span(
^^^^^^^^^
File "/usr/local/lib/python3.12/contextlib.py", line 158, in __exit__
self.gen.throw(value)
File "/tmp/.venv/lib/python3.12/site-packages/opentelemetry/trace/__init__.py", line 619, in use_span
span.end()
File "/tmp/.venv/lib/python3.12/site-packages/instana/span/span.py", line 200, in end
self._span_processor.record_span(self._readable_span())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'BatchSpanProcessor' object has no attribute 'record_span'
🤖 Python Version
3.12.0
📃 Provide any additional context for the Bug.
- All relevant packages were installed (see pip freeze output below):
aiohappyeyeballs==2.6.1
aiohttp==3.13.0
aiosignal==1.4.0
annotated-types==0.7.0
anthropic==0.71.0
anyio==4.11.0
attrs==25.4.0
autowrapt==1.0
backoff==2.2.1
certifi==2025.10.5
charset-normalizer==3.4.4
colorama==0.4.6
cuid==0.4
deprecated==1.2.18
distro==1.9.0
docstring-parser==0.17.0
filelock==3.20.0
frozenlist==1.8.0
fsspec==2025.9.0
fysom==2.1.6
googleapis-common-protos==1.70.0
grpcio==1.75.1
h11==0.16.0
hf-xet==1.1.10
httpcore==1.0.9
httpx==0.28.1
huggingface-hub==0.35.3
idna==3.11
importlib-metadata==8.7.0
inflection==0.5.1
instana==3.9.0
jinja2==3.1.6
jiter==0.11.0
markupsafe==3.0.3
monotonic==1.6
multidict==6.7.0
opentelemetry-api==1.38.0
opentelemetry-exporter-otlp==1.38.0
opentelemetry-exporter-otlp-proto-common==1.38.0
opentelemetry-exporter-otlp-proto-grpc==1.38.0
opentelemetry-exporter-otlp-proto-http==1.38.0
opentelemetry-instrumentation==0.59b0
opentelemetry-instrumentation-alephalpha==0.47.3
opentelemetry-instrumentation-anthropic==0.47.3
opentelemetry-instrumentation-bedrock==0.47.3
opentelemetry-instrumentation-chromadb==0.47.3
opentelemetry-instrumentation-cohere==0.47.3
opentelemetry-instrumentation-crewai==0.47.3
opentelemetry-instrumentation-google-generativeai==0.47.3
opentelemetry-instrumentation-groq==0.47.3
opentelemetry-instrumentation-haystack==0.47.3
opentelemetry-instrumentation-lancedb==0.47.3
opentelemetry-instrumentation-langchain==0.47.3
opentelemetry-instrumentation-llamaindex==0.47.3
opentelemetry-instrumentation-logging==0.59b0
opentelemetry-instrumentation-marqo==0.47.3
opentelemetry-instrumentation-mcp==0.47.3
opentelemetry-instrumentation-milvus==0.47.3
opentelemetry-instrumentation-mistralai==0.47.3
opentelemetry-instrumentation-ollama==0.47.3
opentelemetry-instrumentation-openai==0.47.3
opentelemetry-instrumentation-openai-agents==0.47.3
opentelemetry-instrumentation-pinecone==0.47.3
opentelemetry-instrumentation-qdrant==0.47.3
opentelemetry-instrumentation-redis==0.59b0
opentelemetry-instrumentation-replicate==0.47.3
opentelemetry-instrumentation-requests==0.59b0
opentelemetry-instrumentation-sagemaker==0.47.3
opentelemetry-instrumentation-sqlalchemy==0.59b0
opentelemetry-instrumentation-threading==0.59b0
opentelemetry-instrumentation-together==0.47.3
opentelemetry-instrumentation-transformers==0.47.3
opentelemetry-instrumentation-urllib3==0.59b0
opentelemetry-instrumentation-vertexai==0.47.3
opentelemetry-instrumentation-watsonx==0.47.3
opentelemetry-instrumentation-weaviate==0.47.3
opentelemetry-instrumentation-writer==0.47.3
opentelemetry-proto==1.38.0
opentelemetry-sdk==1.38.0
opentelemetry-semantic-conventions==0.59b0
opentelemetry-semantic-conventions-ai==0.4.13
opentelemetry-util-http==0.59b0
packaging==25.0
posthog==3.25.0
propcache==0.4.1
protobuf==6.33.0
pydantic==2.12.2
pydantic-core==2.41.4
python-dateutil==2.9.0.post0
pyyaml==6.0.3
regex==2025.9.18
requests==2.32.5
setuptools==80.9.0
six==1.17.0
sniffio==1.3.1
tenacity==9.1.2
tiktoken==0.12.0
tokenizers==0.22.1
tqdm==4.67.1
traceloop-sdk==0.47.3
typing-extensions==4.15.0
typing-inspection==0.4.2
urllib3==2.5.0
wrapt==1.17.3
yarl==1.22.0
zipp==3.23.0
Script used for reproduction:
#!/usr/bin/env python3
# traceloop_test.py - Minimalist Python script for sending traces to Instana
# Using direct OpenTelemetry APIs instead of Traceloop decorators
import time
import logging
import os
# import instana
from traceloop.sdk import Traceloop
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor, ConsoleSpanExporter
from opentelemetry.sdk.resources import SERVICE_NAME, Resource
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
# os.environ["AUTOWRAPT_BOOTSTRAP"] = "instana"
os.environ["TRACELOOP_BASE_URL"] = "http://instana-agent.instana-agent.svc.cluster.local:4318"
def setup_minimal_tracing():
"""Setup minimal tracing configuration avoiding metrics issues."""
# Create resource
resource = Resource.create({
SERVICE_NAME: "llm-tracing",
"service.version": "0.1.0.556-develop"
})
# Setup tracer provider
provider = TracerProvider(resource=resource)
# Console exporter for local verification
console_processor = BatchSpanProcessor(ConsoleSpanExporter())
provider.add_span_processor(console_processor)
# OTLP gRPC exporter for traces only
try:
trace_exporter = OTLPSpanExporter(
endpoint="http://instana-agent.instana-agent.svc.cluster.local:4318",
insecure=True,
timeout=10
)
otlp_processor = BatchSpanProcessor(trace_exporter)
provider.add_span_processor(otlp_processor)
logger.info("OTLP trace exporter configured")
except Exception as e:
logger.warning(f" OTLP setup failed: {e}")
# Set tracer provider globally
trace.set_tracer_provider(provider)
# Initialize Traceloop (minimal configuration)
Traceloop.init(app_name="llm-tracing")
return trace.get_tracer(__name__)
def main():
"""Main execution with minimal configuration."""
logger.info("Starting minimal Traceloop + Instana integration")
tracer = setup_minimal_tracing()
# Simple test workflow
with tracer.start_as_current_span("test_banking_workflow") as span:
span.set_attribute("workflow.type", "minimal_test")
span.set_attribute("service.name", "yoda-agent-service")
# Simulate operations
with tracer.start_as_current_span("auth_operation") as auth_span:
auth_span.set_attribute("auth.method", "oauth2")
time.sleep(0.05)
logger.info("Auth operation completed")
with tracer.start_as_current_span("llm_operation") as llm_span:
llm_span.set_attribute("llm.model", "claude-v2")
llm_span.set_attribute("llm.tokens", 150)
time.sleep(0.1)
logger.info("LLM operation completed")
span.set_attribute("workflow.success", True)
logger.info("Test workflow completed")
# Allow trace export
time.sleep(3)
logger.info("Minimal integration test completed")
if __name__ == "__main__":
main()
👀 Have you spent some time to check if this bug has been raised before?
- [x] I checked and didn't find similar issue
Are you willing to submit PR?
No response
Additionally, I tried another alternative approach
Script Used
cat << 'EOF' > traceloop_test.py
#!/usr/bin/env python3
# traceloop_test.py - Minimalist Python script for sending traces to Instana
# Using Traceloop SDK for simplified GenAI observability
import os
import time
import uuid
from datetime import datetime
# Import Traceloop components
from traceloop.sdk import Traceloop
from traceloop.sdk.decorators import workflow, task
# Configuration
USER_ID = "MphoMphego"
SERVICE_NAME_VALUE = "traceloop-py-MphoMphego"
ENDPOINT = "http://instana-agent.instana-agent.svc.cluster.local:4318/v1/traces"
def log(level, message):
"""Simple logging function"""
print(f"{datetime.now().strftime('%Y-%m-%d %H:%M:%S')} [{level}] - {message}")
def setup_tracing():
"""Set up Traceloop tracing with OTLP HTTP exporter for Instana"""
log("INFO", f"Setting up Traceloop tracing for {SERVICE_NAME_VALUE}")
# Initialize Traceloop with configuration
Traceloop.init(
app_name=SERVICE_NAME_VALUE,
api_endpoint=ENDPOINT,
disable_batch=False, # Use batching for better performance
resource_attributes={
"user.id": USER_ID,
}
)
log("INFO", "Traceloop initialized successfully")
@task(name="database.query")
def database_query():
"""Simulate database query operation"""
log("INFO", "Executing database.query")
time.sleep(0.2)
return {"status": "success", "rows": 42}
@task(name="llm.inference")
def llm_inference():
"""Simulate LLM inference operation"""
log("INFO", "Executing llm.inference")
time.sleep(0.2)
return {"status": "success", "tokens": 150}
@task(name="embedding.vector")
def embedding_vector():
"""Simulate embedding vector operation"""
log("INFO", "Executing embedding.vector")
time.sleep(0.2)
return {"status": "success", "dimensions": 1536}
@task(name="rag.retrieval")
def rag_retrieval():
"""Simulate RAG retrieval operation"""
log("INFO", "Executing rag.retrieval")
time.sleep(0.2)
return {"status": "success", "documents": 5}
@task(name="test.verification")
def test_verification():
"""Simulate test verification operation"""
log("INFO", "Executing test.verification")
time.sleep(0.2)
return {"status": "success", "verified": True}
@workflow(name="test_workflow")
def run_test():
"""Run simulated operations using Traceloop decorators"""
trace_id = uuid.uuid4().hex
log("INFO", f"Starting test with trace ID: {trace_id}")
start_time = time.time()
# Operations to simulate
operations = [
"database.query",
"llm.inference",
"embedding.vector",
"rag.retrieval",
"test.verification"
]
log("INFO", f"Simulating operations: {', '.join(operations)}")
# Set workflow attributes using Traceloop's association
Traceloop.set_association_properties({
"conversation_id": trace_id,
"user_id": USER_ID
})
# Run each operation
results = []
log("INFO", "Simulating database.query")
results.append(database_query())
time.sleep(0.1)
log("INFO", "Simulating llm.inference")
results.append(llm_inference())
time.sleep(0.1)
log("INFO", "Simulating embedding.vector")
results.append(embedding_vector())
time.sleep(0.1)
log("INFO", "Simulating rag.retrieval")
results.append(rag_retrieval())
time.sleep(0.1)
log("INFO", "Simulating test.verification")
results.append(test_verification())
time.sleep(0.1)
duration = int((time.time() - start_time) * 1000)
return duration, results
def report(duration):
"""Generate a report of the test run"""
print("\n==== INSTANA TRACE SUMMARY ====")
print(f"Service: {SERVICE_NAME_VALUE}")
print(f"User: {USER_ID}")
print(f"Duration: {duration}ms")
print("")
print("Operations: workflow.execution, database.query, llm.inference, embedding.vector, rag.retrieval, test.verification")
print(f"OTLP HTTP endpoint: {ENDPOINT}")
print("")
print(f"🔍 Search for: {SERVICE_NAME_VALUE} in Instana UI")
print("==================================")
def main():
"""Main function to execute the trace test"""
setup_tracing()
duration, results = run_test()
report(duration)
log("INFO", f"Test completed in {duration}ms with {len(results)} operations")
# Allow time for span export before exit
log("INFO", "Waiting for spans to be exported...")
time.sleep(5)
if __name__ == "__main__":
main()
EOF
# Make the script executable
chmod +x traceloop_test.py
echo "Updated Python script traceloop_test.py created successfully!"
echo "Details:"
echo "- Date/time: 2025-10-17 11:14:46 (UTC)"
echo "- User: MphoMphego"
echo "- Service name: traceloop-py-MphoMphego"
echo ""
echo "To run the script:"
echo " pip install traceloop-sdk"
echo " ./traceloop_test.py"
Logs
2025-10-17 11:21:19 [INFO] - Setting up Traceloop tracing for traceloop-py-MphoMphego
2025-10-17 11:21:20 [INFO] - Traceloop initialized successfully
2025-10-17 11:21:20 [INFO] - Starting test with trace ID: a8756587e7334e1198c44a8a1235cc89
2025-10-17 11:21:20 [INFO] - Simulating operations: database.query, llm.inference, embedding.vector, rag.retrieval, test.verification
2025-10-17 11:21:20 [INFO] - Simulating database.query
2025-10-17 11:21:20 [INFO] - Executing database.query
Traceback (most recent call last):
File "/tmp/.venv/lib/python3.12/site-packages/traceloop/sdk/decorators/base.py", line 272, in sync_wrap
res = fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "/tmp/./traceloop_test.py", line 104, in run_test
results.append(database_query())
^^^^^^^^^^^^^^^^
File "/tmp/.venv/lib/python3.12/site-packages/traceloop/sdk/decorators/base.py", line 284, in sync_wrap
_cleanup_span(span, ctx_token)
File "/tmp/.venv/lib/python3.12/site-packages/traceloop/sdk/decorators/base.py", line 207, in _cleanup_span
span.end()
File "/opt/instana/instrumentation/python/3.12.0/instana/span/span.py", line 200, in end
self._span_processor.record_span(self._readable_span())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'BatchSpanProcessor' object has no attribute 'record_span'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/tmp/./traceloop_test.py", line 151, in <module>
main()
File "/tmp/./traceloop_test.py", line 142, in main
duration, results = run_test()
^^^^^^^^^^
File "/tmp/.venv/lib/python3.12/site-packages/traceloop/sdk/decorators/base.py", line 276, in sync_wrap
_cleanup_span(span, ctx_token)
File "/tmp/.venv/lib/python3.12/site-packages/traceloop/sdk/decorators/base.py", line 207, in _cleanup_span
span.end()
File "/opt/instana/instrumentation/python/3.12.0/instana/span/span.py", line 200, in end
self._span_processor.record_span(self._readable_span())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'BatchSpanProcessor' object has no attribute 'record_span'
Packages installed
VirtualEnv (/tmp/.venv)
aiohappyeyeballs==2.6.1
aiohttp==3.13.0
aiosignal==1.4.0
annotated-types==0.7.0
anthropic==0.71.0
anyio==4.11.0
attrs==25.4.0
backoff==2.2.1
certifi==2025.10.5
charset-normalizer==3.4.4
colorama==0.4.6
cuid==0.4
deprecated==1.2.18
distro==1.9.0
docstring-parser==0.17.0
filelock==3.20.0
frozenlist==1.8.0
fsspec==2025.9.0
googleapis-common-protos==1.70.0
grpcio==1.75.1
h11==0.16.0
hf-xet==1.1.10
httpcore==1.0.9
httpx==0.28.1
huggingface-hub==0.35.3
idna==3.11
importlib-metadata==8.7.0
inflection==0.5.1
jinja2==3.1.6
jiter==0.11.0
markupsafe==3.0.3
monotonic==1.6
multidict==6.7.0
opentelemetry-api==1.38.0
opentelemetry-exporter-otlp==1.38.0
opentelemetry-exporter-otlp-proto-common==1.38.0
opentelemetry-exporter-otlp-proto-grpc==1.38.0
opentelemetry-exporter-otlp-proto-http==1.38.0
opentelemetry-instrumentation==0.59b0
opentelemetry-instrumentation-alephalpha==0.47.3
opentelemetry-instrumentation-anthropic==0.47.3
opentelemetry-instrumentation-bedrock==0.47.3
opentelemetry-instrumentation-chromadb==0.47.3
opentelemetry-instrumentation-cohere==0.47.3
opentelemetry-instrumentation-crewai==0.47.3
opentelemetry-instrumentation-google-generativeai==0.47.3
opentelemetry-instrumentation-groq==0.47.3
opentelemetry-instrumentation-haystack==0.47.3
opentelemetry-instrumentation-lancedb==0.47.3
opentelemetry-instrumentation-langchain==0.47.3
opentelemetry-instrumentation-llamaindex==0.47.3
opentelemetry-instrumentation-logging==0.59b0
opentelemetry-instrumentation-marqo==0.47.3
opentelemetry-instrumentation-mcp==0.47.3
opentelemetry-instrumentation-milvus==0.47.3
opentelemetry-instrumentation-mistralai==0.47.3
opentelemetry-instrumentation-ollama==0.47.3
opentelemetry-instrumentation-openai==0.47.3
opentelemetry-instrumentation-openai-agents==0.47.3
opentelemetry-instrumentation-pinecone==0.47.3
opentelemetry-instrumentation-qdrant==0.47.3
opentelemetry-instrumentation-redis==0.59b0
opentelemetry-instrumentation-replicate==0.47.3
opentelemetry-instrumentation-requests==0.59b0
opentelemetry-instrumentation-sagemaker==0.47.3
opentelemetry-instrumentation-sqlalchemy==0.59b0
opentelemetry-instrumentation-threading==0.59b0
opentelemetry-instrumentation-together==0.47.3
opentelemetry-instrumentation-transformers==0.47.3
opentelemetry-instrumentation-urllib3==0.59b0
opentelemetry-instrumentation-vertexai==0.47.3
opentelemetry-instrumentation-watsonx==0.47.3
opentelemetry-instrumentation-weaviate==0.47.3
opentelemetry-instrumentation-writer==0.47.3
opentelemetry-proto==1.38.0
opentelemetry-sdk==1.38.0
opentelemetry-semantic-conventions==0.59b0
opentelemetry-semantic-conventions-ai==0.4.13
opentelemetry-util-http==0.59b0
packaging==25.0
posthog==3.25.0
propcache==0.4.1
protobuf==6.33.0
pydantic==2.12.2
pydantic-core==2.41.4
python-dateutil==2.9.0.post0
pyyaml==6.0.3
requests==2.32.5
six==1.17.0
sniffio==1.3.1
tenacity==9.1.2
tokenizers==0.22.1
tqdm==4.67.1
traceloop-sdk==0.47.3
typing-extensions==4.15.0
typing-inspection==0.4.2
urllib3==2.5.0
wrapt==1.17.3
yarl==1.22.0
zipp==3.23.0
System
absolufy-imports==0.3.1
aiohappyeyeballs==2.6.1
aiohttp==3.13.0
aiosignal==1.4.0
alembic==1.17.0
annotated-types==0.7.0
anyio==4.11.0
APScheduler==3.11.0
attrs==25.4.0
Authlib==1.6.5
autocommand==2.2.2
autowrapt==1.0
backoff==2.2.1
backports.tarfile==1.2.0
boto3==1.40.53
botocore==1.40.53
cachetools==6.2.1
certifi==2025.10.5
cffi==2.0.0
charset-normalizer==3.4.4
click==8.3.0
cloudpickle==3.1.1
cryptography==46.0.3
distro==1.9.0
dnspython==2.8.0
docstring_parser==0.17.0
ecdsa==0.19.1
email-validator==2.3.0
fastapi==0.119.0
fastapi-sso==0.18.0
fastuuid==0.13.5
filelock==3.20.0
frozenlist==1.8.0
fsspec==2025.9.0
fysom==2.1.6
google-adk==1.16.0
google-api-core==2.26.0
google-api-python-client==2.184.0
google-auth==2.41.1
google-auth-httplib2==0.2.0
google-cloud-aiplatform==1.121.0
google-cloud-appengine-logging==1.6.2
google-cloud-audit-log==0.3.3
google-cloud-bigquery==3.38.0
google-cloud-bigtable==2.33.0
google-cloud-core==2.4.3
google-cloud-discoveryengine==0.13.12
google-cloud-logging==3.12.1
google-cloud-monitoring==2.28.0
google-cloud-resource-manager==1.14.2
google-cloud-secret-manager==2.25.0
google-cloud-spanner==3.58.0
google-cloud-speech==2.33.0
google-cloud-storage==2.19.0
google-cloud-trace==1.16.2
google-crc32c==1.7.1
google-genai==1.45.0
google-resumable-media==2.7.2
googleapis-common-protos==1.70.0
graphviz==0.21
greenlet==3.2.4
grpc-google-iam-v1==0.14.3
grpc-interceptor==0.15.4
grpcio==1.75.1
grpcio-status==1.71.2
h11==0.16.0
h2==4.3.0
hf-xet==1.1.10
hpack==4.1.0
httpcore==1.0.9
httplib2==0.31.0
httptools==0.7.1
httpx==0.28.1
httpx-sse==0.4.3
huggingface-hub==0.35.3
hyperframe==6.1.0
idna==3.11
importlib_metadata==8.7.0
inflect==7.3.1
instana==3.8.2
jaraco.collections==5.1.0
jaraco.context==5.3.0
jaraco.functools==4.0.1
jaraco.text==3.12.1
Jinja2==3.1.6
jiter==0.11.0
jmespath==1.0.1
jsonschema==4.25.1
jsonschema-specifications==2025.9.1
langfuse==3.6.2
litellm==1.78.2
Mako==1.3.10
MarkupSafe==3.0.3
mcp==1.17.0
mem0ai==1.0.0
more-itertools==10.3.0
multidict==6.7.0
neo4j==6.0.2
numpy==2.3.4
oauthlib==3.3.1
openai==1.99.5
opentelemetry-api==1.37.0
opentelemetry-exporter-gcp-logging==1.10.0a0
opentelemetry-exporter-gcp-monitoring==1.10.0a0
opentelemetry-exporter-gcp-trace==1.10.0
opentelemetry-exporter-otlp-proto-common==1.37.0
opentelemetry-exporter-otlp-proto-http==1.37.0
opentelemetry-proto==1.37.0
opentelemetry-resourcedetector-gcp==1.10.0a0
opentelemetry-sdk==1.37.0
opentelemetry-semantic-conventions==0.58b0
orjson==3.11.3
packaging==25.0
platformdirs==4.2.2
portalocker==3.2.0
posthog==6.7.7
propcache==0.4.1
proto-plus==1.26.1
protobuf==5.29.5
psycopg==3.2.10
psycopg-binary==3.2.10
psycopg-pool==3.2.6
psycopg2-binary==2.9.11
pyasn1==0.6.1
pyasn1_modules==0.4.2
pycparser==2.23
pydantic==2.12.2
pydantic-settings==2.11.0
pydantic_core==2.41.4
PyJWT==2.10.1
pyparsing==3.2.5
python-dateutil==2.9.0.post0
python-dotenv==1.1.1
python-jose==3.5.0
python-multipart==0.0.20
pytz==2025.2
PyYAML==6.0.3
qdrant-client==1.15.1
referencing==0.37.0
regex==2025.9.18
requests==2.32.5
rpds-py==0.27.1
rsa==4.9.1
s3transfer==0.14.0
setuptools==80.9.0
shapely==2.1.2
six==1.17.0
sniffio==1.3.1
SQLAlchemy==2.0.44
sqlalchemy-spanner==1.17.0
sqlparse==0.5.3
sse-starlette==3.0.2
starlette==0.48.0
tenacity==8.5.0
tiktoken==0.12.0
tokenizers==0.22.1
tomli==2.0.1
tqdm==4.67.1
typeguard==4.3.0
typing-inspection==0.4.2
typing_extensions==4.15.0
tzlocal==5.3.1
uritemplate==4.2.0
urllib3==2.5.0
uv==0.9.3
uvicorn==0.37.0
uvloop==0.21.0
watchdog==6.0.0
watchfiles==1.1.1
websockets==15.0.1
wheel==0.45.1
wrapt==1.17.3
yarl==1.22.0
zipp==3.23.0
@adharshctr can you help check the issue? I think we did not test the latest release yet. Thanks
@adharshctr can you help check the issue? I think we did not test the latest release yet. Thanks
Sure
Hi @mmphego I recreated the same environment and ran the first application after a minor refactor. The traces are now visible in Instana.
When you initialize OpenTelemetry first (or manually call Resource.create()), it sets a global resource that includes the service.name and service.version. Then, when you later call, Traceloop.init(app_name="llm-tracing"), Traceloop detects that a global tracer provider already exists and won’t override it. As a result, the service.name from OpenTelemetry takes precedence ,not the one from Traceloop.
Could you please try running the application below, exporting the traces to port 4317? Also verify that the provided hostname is correct
#!/usr/bin/env python3
# traceloop_test.py - Minimalist Python script for sending traces to Instana
# Using direct OpenTelemetry APIs instead of Traceloop decorators
import time
import logging
import os
# import instana
from traceloop.sdk import Traceloop
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor, ConsoleSpanExporter
from opentelemetry.sdk.resources import SERVICE_NAME, Resource
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
Traceloop.init(app_name="llm-tracing")
# os.environ["AUTOWRAPT_BOOTSTRAP"] = "instana"
os.environ["TRACELOOP_BASE_URL"] = "http://instana-agent.instana-agent.svc.cluster.local:4318"
def setup_minimal_tracing():
"""Setup minimal tracing configuration avoiding metrics issues."""
# Create resource
resource = Resource.create({
SERVICE_NAME: "llm-tracing",
"service.version": "0.1.0.556-develop"
})
# Setup tracer provider
provider = TracerProvider(resource=resource)
# Console exporter for local verification
console_processor = BatchSpanProcessor(ConsoleSpanExporter())
provider.add_span_processor(console_processor)
# OTLP gRPC exporter for traces only
try:
trace_exporter = OTLPSpanExporter(
endpoint="http://instana-agent.instana-agent.svc.cluster.local:4318",
insecure=True,
timeout=10
)
otlp_processor = BatchSpanProcessor(trace_exporter)
provider.add_span_processor(otlp_processor)
logger.info("OTLP trace exporter configured")
except Exception as e:
logger.warning(f" OTLP setup failed: {e}")
# Set tracer provider globally
trace.set_tracer_provider(provider)
return trace.get_tracer(__name__)
def main():
"""Main execution with minimal configuration."""
logger.info("Starting minimal Traceloop + Instana integration")
tracer = setup_minimal_tracing()
# Simple test workflow
with tracer.start_as_current_span("test_banking_workflow") as span:
span.set_attribute("workflow.type", "minimal_test")
span.set_attribute("service.name", "yoda-agent-service")
# Simulate operations
with tracer.start_as_current_span("auth_operation") as auth_span:
auth_span.set_attribute("auth.method", "oauth2")
time.sleep(0.05)
logger.info("Auth operation completed")
with tracer.start_as_current_span("llm_operation") as llm_span:
llm_span.set_attribute("llm.model", "claude-v2")
llm_span.set_attribute("llm.tokens", 150)
time.sleep(0.1)
logger.info("LLM operation completed")
span.set_attribute("workflow.success", True)
logger.info("Test workflow completed")
# Allow trace export
time.sleep(3)
logger.info("Minimal integration test completed")
if __name__ == "__main__":
main()
OUTPUT
@adharshctr can you highlight the changes you have made? Thanks
Initialized the traceloop on the Top.
Hi @mmphego I recreated the same environment and ran the first application after a minor refactor. The traces are now visible in Instana.
When you initialize OpenTelemetry first (or manually call Resource.create()), it sets a global resource that includes the service.name and service.version. Then, when you later call, Traceloop.init(app_name="llm-tracing"), Traceloop detects that a global tracer provider already exists and won’t override it. As a result, the service.name from OpenTelemetry takes precedence ,not the one from Traceloop.
Could you please try running the application below, exporting the traces to port 4317? Also verify that the provided hostname is correct
I attempted to replicate the issue described in your bug report. Here's my step-by-step investigation:
1. Sanity Check - Direct Instana Agent Connectivity
First, I performed a sanity check to ensure that I could successfully send traces directly to the Instana agent endpoint using a simple curl command:
TRACE_ID=$(openssl rand -hex 16); echo "Trace ID: $TRACE_ID"; curl -X POST -H "Content-Type: application/json" -d '{"resourceSpans":[{"resource":{"attributes":[{"key":"service.name","value":{"stringValue":"quicktest-MphoMphego"}},{"key":"user","value":{"stringValue":"MphoMphego"}}]},"scopeSpans":[{"spans":[{"traceId":"'$TRACE_ID'","spanId":"'$(openssl rand -hex 8)'","name":"quick-test","startTimeUnixNano":"'$(date +%s%N)'","endTimeUnixNano":"'$(($(date +%s%N) + 10000000))'","status":{"code":1}}]}]}]}' http://instana-agent.instana-agent.svc.cluster.local:4318/v1/traces && echo -e "\n🔍 Search for: quicktest-MphoMphego or $TRACE_ID"
Click on gif below
Result: ✅ Successfully verified that traces are received and visible in Instana.
2. Package Installation
Next, I installed the required packages as specified in your bug report:
uname -srmo
cd /tmp
uv --no-cache venv
source .venv/bin/activate
uv --no-cache pip install traceloop-sdk
Output
Click to expand code
Linux 6.1.147-172.266.amzn2023.x86_64 x86_64 GNU/Linux
Using CPython 3.12.12 interpreter at: /usr/local/bin/python
Creating virtual environment at: .venv
Activate with: source .venv/bin/activate
Resolved 103 packages in 3.51s
Built cuid==0.4
Prepared 103 packages in 1.46s
Installed 103 packages in 256ms
+ aiohappyeyeballs==2.6.1
+ aiohttp==3.13.1
+ aiosignal==1.4.0
+ annotated-types==0.7.0
+ anthropic==0.71.0
+ anyio==4.11.0
+ attrs==25.4.0
+ backoff==2.2.1
+ certifi==2025.10.5
+ charset-normalizer==3.4.4
+ click==8.3.0
+ colorama==0.4.6
+ cuid==0.4
+ deprecated==1.2.18
+ distro==1.9.0
+ docstring-parser==0.17.0
+ filelock==3.20.0
+ frozenlist==1.8.0
+ fsspec==2025.9.0
+ googleapis-common-protos==1.71.0
+ grpcio==1.76.0
+ h11==0.16.0
+ hf-xet==1.2.0
+ httpcore==1.0.9
+ httpx==0.28.1
+ huggingface-hub==1.0.0
+ idna==3.11
+ importlib-metadata==8.7.0
+ inflection==0.5.1
+ jinja2==3.1.6
+ jiter==0.11.1
+ markupsafe==3.0.3
+ monotonic==1.6
+ multidict==6.7.0
+ opentelemetry-api==1.38.0
+ opentelemetry-exporter-otlp==1.38.0
+ opentelemetry-exporter-otlp-proto-common==1.38.0
+ opentelemetry-exporter-otlp-proto-grpc==1.38.0
+ opentelemetry-exporter-otlp-proto-http==1.38.0
+ opentelemetry-instrumentation==0.59b0
+ opentelemetry-instrumentation-alephalpha==0.47.5
+ opentelemetry-instrumentation-anthropic==0.47.5
+ opentelemetry-instrumentation-bedrock==0.47.5
+ opentelemetry-instrumentation-chromadb==0.47.5
+ opentelemetry-instrumentation-cohere==0.47.5
+ opentelemetry-instrumentation-crewai==0.47.5
+ opentelemetry-instrumentation-google-generativeai==0.47.5
+ opentelemetry-instrumentation-groq==0.47.5
+ opentelemetry-instrumentation-haystack==0.47.5
+ opentelemetry-instrumentation-lancedb==0.47.5
+ opentelemetry-instrumentation-langchain==0.47.5
+ opentelemetry-instrumentation-llamaindex==0.47.5
+ opentelemetry-instrumentation-logging==0.59b0
+ opentelemetry-instrumentation-marqo==0.47.5
+ opentelemetry-instrumentation-mcp==0.47.5
+ opentelemetry-instrumentation-milvus==0.47.5
+ opentelemetry-instrumentation-mistralai==0.47.5
+ opentelemetry-instrumentation-ollama==0.47.5
+ opentelemetry-instrumentation-openai==0.47.5
+ opentelemetry-instrumentation-openai-agents==0.47.5
+ opentelemetry-instrumentation-pinecone==0.47.5
+ opentelemetry-instrumentation-qdrant==0.47.5
+ opentelemetry-instrumentation-redis==0.59b0
+ opentelemetry-instrumentation-replicate==0.47.5
+ opentelemetry-instrumentation-requests==0.59b0
+ opentelemetry-instrumentation-sagemaker==0.47.5
+ opentelemetry-instrumentation-sqlalchemy==0.59b0
+ opentelemetry-instrumentation-threading==0.59b0
+ opentelemetry-instrumentation-together==0.47.5
+ opentelemetry-instrumentation-transformers==0.47.5
+ opentelemetry-instrumentation-urllib3==0.59b0
+ opentelemetry-instrumentation-vertexai==0.47.5
+ opentelemetry-instrumentation-watsonx==0.47.5
+ opentelemetry-instrumentation-weaviate==0.47.5
+ opentelemetry-instrumentation-writer==0.47.5
+ opentelemetry-proto==1.38.0
+ opentelemetry-sdk==1.38.0
+ opentelemetry-semantic-conventions==0.59b0
+ opentelemetry-semantic-conventions-ai==0.4.13
+ opentelemetry-util-http==0.59b0
+ packaging==25.0
+ posthog==3.25.0
+ propcache==0.4.1
+ protobuf==6.33.0
+ pydantic==2.12.3
+ pydantic-core==2.41.4
+ python-dateutil==2.9.0.post0
+ pyyaml==6.0.3
+ requests==2.32.5
+ shellingham==1.5.4
+ six==1.17.0
+ sniffio==1.3.1
+ tenacity==9.1.2
+ tokenizers==0.22.1
+ tqdm==4.67.1
+ traceloop-sdk==0.47.5
+ typer-slim==0.20.0
+ typing-extensions==4.15.0
+ typing-inspection==0.4.2
+ urllib3==2.5.0
+ wrapt==1.17.3
+ yarl==1.22.0
+ zipp==3.23.0
3. Script Execution and Issue Reproduction
I copied and executed the provided script exactly as written, however, I encountered additional issues beyond the reported AttributeError. The script itself contains several problems that prevent it from running successfully:
Issues Identified:
- Unsupported Parameter: The script uses an insecure argument in the OTLPSpanExporter constructor, which is not supported by the HTTP trace exporter class. This parameter only exists in the gRPC exporter.
- SSL Certificate Verification Errors: Traceloop SDK attempts to send analytics data to PostHog (us.i.posthog.com) by default, which fails due to SSL certificate verification issues in containerized environments.
Error Details:
Click to expand code
:/tmp$ python traceloop_test.py
INFO:__main__:Starting minimal Traceloop + Instana integration
INFO:__main__:OTLP trace exporter configured
WARNING:opentelemetry.trace:Overriding of current TracerProvider is not allowed
Current Trace ID (integer): 37427765274862389586295400368467639057
Current Trace ID (hex): 1c28534cb8d79c11534bd34e0f107b11
INFO:__main__:Auth operation completed
INFO:__main__:LLM operation completed
INFO:__main__:Test workflow completed
WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=2, read=2, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1010)'))': /batch/
WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=2, read=2, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1010)'))': /batch/
INFO:backoff:Backing off send_request(...) for 0.8s (requests.exceptions.SSLError: HTTPSConnectionPool(host='us.i.posthog.com', port=443): Max retries exceeded with url: /batch/ (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1010)'))))
WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=2, read=2, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1010)'))': /batch/
WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=2, read=2, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1010)'))': /batch/
INFO:__main__:Minimal integration test completed
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1761652687.736658 206 ssl_transport_security.cc:1884] Handshake failed with error SSL_ERROR_SSL: error:100000f7:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER: Invalid certificate verification context
WARNING:opentelemetry.exporter.otlp.proto.grpc.exporter:Transient error StatusCode.UNAVAILABLE encountered while exporting traces to instana-agent.instana-agent:4317, retrying in 0.82s.
INFO:backoff:Backing off send_request(...) for 0.4s (requests.exceptions.SSLError: HTTPSConnectionPool(host='us.i.posthog.com', port=443): Max retries exceeded with url: /batch/ (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1010)'))))
WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=2, read=2, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1010)'))': /batch/
WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=2, read=2, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1010)'))': /batch/
INFO:backoff:Backing off send_request(...) for 1.6s (requests.exceptions.SSLError: HTTPSConnectionPool(host='us.i.posthog.com', port=443): Max retries exceeded with url: /batch/ (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1010)'))))
I0000 00:00:1761652689.101924 208 ssl_transport_security.cc:1884] Handshake failed with error SSL_ERROR_SSL: error:100000f7:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER: Invalid certificate verification context
WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=2, read=2, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1010)'))': /batch/
WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=2, read=2, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1010)'))': /batch/
ERROR:backoff:Giving up send_request(...) after 4 tries (requests.exceptions.SSLError: HTTPSConnectionPool(host='us.i.posthog.com', port=443): Max retries exceeded with url: /batch/ (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1010)'))))
I0000 00:00:1761652694.101906 203 ssl_transport_security.cc:1884] Handshake failed with error SSL_ERROR_SSL: error:100000f7:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER: Invalid certificate verification context
ERROR:opentelemetry.exporter.otlp.proto.grpc.exporter:Failed to export traces to instana-agent.instana-agent:4317, error code: StatusCode.UNAVAILABLE
The GIF demonstrates the execution and highlights the specific errors encountered:
https://jumpshare.com/s/JfTaYOJYSNaMQXSgSyyD
Resolution Applied:
To properly test the original reported issue, I had to first fix these script problems:
- Removed unsupported insecure parameter from OTLPSpanExporter
- Disabled Traceloop telemetry by setting telemetry_enabled=False in Traceloop.init()
- Added proper SSL handling with custom session configuration
- Added missing imports and environment variable settings
After applying these fixes, I was able to proceed with reproducing the results you got
Click to expand code
# Set environment variables
export TRACELOOP_TELEMETRY=false
export OTEL_EXPORTER_OTLP_TRACES_CERTIFICATE=false
export TRACELOOP_BASE_URL="http://instana-agent.instana-agent.svc.cluster.local:4318"
export TRACELOOP_DISABLE_ANALYTICS=true # Disable PostHog Analytics (Traceloop Telemetry)
export POSTHOG_DISABLED=true
export POSTHOG_HOST=""
export POSTHOG_PROJECT_API_KEY=""
# SSL/TLS settings
export PYTHONHTTPSVERIFY=0
export SSL_VERIFY=false
export CURL_CA_BUNDLE=/app/ca-bundled.pem
export REQUESTS_CA_BUNDLE=/app/ca-bundled.pem
export SSL_CERT_FILE=/app/ca-bundled.pem
# OTLP settings
export OTEL_EXPORTER_OTLP_TRACES_INSECURE=true
# Disable urllib3 warnings
export PYTHONWARNINGS="ignore:Unverified HTTPS request"
cat << 'EOF' > traceloop_test.py
#!/usr/bin/env python3
# traceloop_test.py - Minimalist Python script for sending traces to Instana
# Using direct OpenTelemetry APIs instead of Traceloop decorators
import time
import logging
import os
import requests
import urllib3
# Set environment variables at runtime and disable SSL warnings
os.environ.update({
"TRACELOOP_TELEMETRY": "false",
"PYTHONHTTPSVERIFY": "0",
"OTEL_EXPORTER_OTLP_TRACES_CERTIFICATE": "false"
})
urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
# import instana
from traceloop.sdk import Traceloop
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor, ConsoleSpanExporter
from opentelemetry.sdk.resources import SERVICE_NAME, Resource
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
Traceloop.init(app_name="llm-tracing", telemetry_enabled=False)
# os.environ["AUTOWRAPT_BOOTSTRAP"] = "instana"
os.environ["TRACELOOP_BASE_URL"] = "http://instana-agent.instana-agent.svc.cluster.local:4318"
def setup_minimal_tracing():
"""Setup minimal tracing configuration avoiding metrics issues."""
# Create resource
resource = Resource.create({
SERVICE_NAME: "llm-tracing",
"service.version": "0.1.0.556-develop"
})
# Setup tracer provider
provider = TracerProvider(resource=resource)
# Console exporter for local verification
console_processor = BatchSpanProcessor(ConsoleSpanExporter())
provider.add_span_processor(console_processor)
# OTLP HTTP exporter for traces only with SSL verification disabled
try:
# Create a session with SSL verification disabled
session = requests.Session()
session.verify = False
trace_exporter = OTLPSpanExporter(
endpoint="http://instana-agent.instana-agent.svc.cluster.local:4318/v1/traces",
timeout=10,
session=session
)
otlp_processor = BatchSpanProcessor(trace_exporter)
provider.add_span_processor(otlp_processor)
logger.info("OTLP trace exporter configured")
except Exception as e:
logger.warning(f" OTLP setup failed: {e}")
# Set tracer provider globally
trace.set_tracer_provider(provider)
return trace.get_tracer(__name__)
def main():
"""Main execution with minimal configuration."""
logger.info("Starting minimal Traceloop + Instana integration")
tracer = setup_minimal_tracing()
# Simple test workflow
with tracer.start_as_current_span("test_banking_workflow") as span:
# Get the span context
span_context = span.get_span_context()
# Access the trace ID from the span context
trace_id = span_context.trace_id
# Trace IDs are typically 16-byte (128-bit) integers.
# You might want to format it for display, e.g., as a hex string.
trace_id_hex = f"{trace_id:032x}"
print(f"Current Trace ID (integer): {trace_id}")
print(f"Current Trace ID (hex): {trace_id_hex}")
print(f"🔍 Search in Instana for trace ID: {trace_id_hex}")
span.set_attribute("workflow.type", "minimal_test")
span.set_attribute("service.name", "test-agent-service")
# Simulate operations
with tracer.start_as_current_span("auth_operation") as auth_span:
auth_span.set_attribute("auth.method", "oauth2")
time.sleep(0.05)
logger.info("Auth operation completed")
with tracer.start_as_current_span("llm_operation") as llm_span:
llm_span.set_attribute("llm.model", "claude-v2")
llm_span.set_attribute("llm.tokens", 150)
time.sleep(0.1)
logger.info("LLM operation completed")
span.set_attribute("workflow.success", True)
logger.info("Test workflow completed")
# Allow trace export
time.sleep(3)
logger.info("Minimal integration test completed")
if __name__ == "__main__":
main()
EOF
echo "Environment variables set and script created successfully!"
echo "To run: python traceloop_test.py"
The GIF demonstrates the execution and highlights the specific errors encountered:
https://jumpshare.com/share/5wye4WnIkNKUbGVfSmTg
Result: ❌ Executed the script successfully but DID NOT receive traces in Instana.
@mmphego Can you try setting OTEL_EXPORTER_OTLP_INSECURE=true to resolve SSL issue ? Also remove insecure from OTLPSpanExporter.
@mmphego Can you try setting OTEL_EXPORTER_OTLP_INSECURE=true to resolve SSL issue ? Also remove insecure from OTLPSpanExporter.
See the rest of the comment above and code used
@mmphego I assume traces work fine with gRPC, and the issue is only with HTTP?
@mmphego I assume traces work fine with gRPC, and the issue is only with HTTP?
Not considering gRPC as it introduces extra complexity and connection management overhead that adds little real-world performance gain for our telemetry use cases. Hence, we are using OTLP over HTTP because it’s simpler, more interoperable, and operationally more reliable across mixed environment.
Can you try to replicate the traceloop implementation in https://github.com/traceloop/openllmetry/issues/3414#issuecomment-3415100087
Or does traceloop behave differently when one isn't using instrumentation and real model invocations?
@mmphego I had tested the application that you had shared. It is working from my end and getting the traces. Could you check whether the port is reachable? curl -v http://instana-agent.instana-agent.svc.cluster.local:4318
@mmphego I had tested the application that you had shared. It is working from my end and getting the traces. Could you check whether the port is reachable? curl -v http://instana-agent.instana-agent.svc.cluster.local:4318
$ curl -v http://instana-agent.instana-agent.svc.cluster.local:4318
* Trying 172.20.156.172:4318...
* Connected to instana-agent.instana-agent.svc.cluster.local (172.20.156.172) port 4318 (#0)
> GET / HTTP/1.1
> Host: instana-agent.instana-agent.svc.cluster.local:4318
> User-Agent: curl/7.88.1
> Accept: */*
>
< HTTP/1.1 404 Not Found
< server: Instana Agent
< content-length: 0
<
* Connection #0 to host instana-agent.instana-agent.svc.cluster.local left intact
$ curl -v http://instana-agent.instana-agent.svc.cluster.local:4318/v1/traces
* Trying 172.20.156.172:4318...
* Connected to instana-agent.instana-agent.svc.cluster.local (172.20.156.172) port 4318 (#0)
> GET /v1/traces HTTP/1.1
> Host: instana-agent.instana-agent.svc.cluster.local:4318
> User-Agent: curl/7.88.1
> Accept: */*
>
< HTTP/1.1 404 Not Found
< server: Instana Agent
< content-length: 0
<
* Connection #0 to host instana-agent.instana-agent.svc.cluster.local left intact
Can we perhaps setup a session and we pair-program? cc: @gyliu513
Could you please update the script as shown below and then try running the application once?
os.environ["TRACELOOP_BASE_URL"] = "http://instana-agent.instana-agent.svc.cluster.local:4318"
Traceloop.init(app_name="llm-tracing", telemetry_enabled=False)
@mmphego I also tested the script below and was able to get the traces. The change I made was updating the ENDPOINT to "http://instana-agent.instana-agent.svc.cluster.local:4318"
cat << 'EOF' > traceloop_test.py
#!/usr/bin/env python3
# traceloop_test.py - Minimalist Python script for sending traces to Instana
# Using Traceloop SDK for simplified GenAI observability
import os
import time
import uuid
from datetime import datetime
# Import Traceloop components
from traceloop.sdk import Traceloop
from traceloop.sdk.decorators import workflow, task
# Configuration
USER_ID = "MphoMphego"
SERVICE_NAME_VALUE = "traceloop-py-MphoMphego"
ENDPOINT = "http://instana-agent.instana-agent.svc.cluster.local:4318"
def log(level, message):
"""Simple logging function"""
print(f"{datetime.now().strftime('%Y-%m-%d %H:%M:%S')} [{level}] - {message}")
def setup_tracing():
"""Set up Traceloop tracing with OTLP HTTP exporter for Instana"""
log("INFO", f"Setting up Traceloop tracing for {SERVICE_NAME_VALUE}")
# Initialize Traceloop with configuration
Traceloop.init(
app_name=SERVICE_NAME_VALUE,
api_endpoint=ENDPOINT,
disable_batch=False, # Use batching for better performance
resource_attributes={
"user.id": USER_ID,
}
)
log("INFO", "Traceloop initialized successfully")
@task(name="database.query")
def database_query():
"""Simulate database query operation"""
log("INFO", "Executing database.query")
time.sleep(0.2)
return {"status": "success", "rows": 42}
@task(name="llm.inference")
def llm_inference():
"""Simulate LLM inference operation"""
log("INFO", "Executing llm.inference")
time.sleep(0.2)
return {"status": "success", "tokens": 150}
@task(name="embedding.vector")
def embedding_vector():
"""Simulate embedding vector operation"""
log("INFO", "Executing embedding.vector")
time.sleep(0.2)
return {"status": "success", "dimensions": 1536}
@task(name="rag.retrieval")
def rag_retrieval():
"""Simulate RAG retrieval operation"""
log("INFO", "Executing rag.retrieval")
time.sleep(0.2)
return {"status": "success", "documents": 5}
@task(name="test.verification")
def test_verification():
"""Simulate test verification operation"""
log("INFO", "Executing test.verification")
time.sleep(0.2)
return {"status": "success", "verified": True}
@workflow(name="test_workflow")
def run_test():
"""Run simulated operations using Traceloop decorators"""
trace_id = uuid.uuid4().hex
log("INFO", f"Starting test with trace ID: {trace_id}")
start_time = time.time()
# Operations to simulate
operations = [
"database.query",
"llm.inference",
"embedding.vector",
"rag.retrieval",
"test.verification"
]
log("INFO", f"Simulating operations: {', '.join(operations)}")
# Set workflow attributes using Traceloop's association
Traceloop.set_association_properties({
"conversation_id": trace_id,
"user_id": USER_ID
})
# Run each operation
results = []
log("INFO", "Simulating database.query")
results.append(database_query())
time.sleep(0.1)
log("INFO", "Simulating llm.inference")
results.append(llm_inference())
time.sleep(0.1)
log("INFO", "Simulating embedding.vector")
results.append(embedding_vector())
time.sleep(0.1)
log("INFO", "Simulating rag.retrieval")
results.append(rag_retrieval())
time.sleep(0.1)
log("INFO", "Simulating test.verification")
results.append(test_verification())
time.sleep(0.1)
duration = int((time.time() - start_time) * 1000)
return duration, results
def report(duration):
"""Generate a report of the test run"""
print("\n==== INSTANA TRACE SUMMARY ====")
print(f"Service: {SERVICE_NAME_VALUE}")
print(f"User: {USER_ID}")
print(f"Duration: {duration}ms")
print("")
print("Operations: workflow.execution, database.query, llm.inference, embedding.vector, rag.retrieval, test.verification")
print(f"OTLP HTTP endpoint: {ENDPOINT}")
print("")
print(f"🔍 Search for: {SERVICE_NAME_VALUE} in Instana UI")
print("==================================")
def main():
"""Main function to execute the trace test"""
setup_tracing()
duration, results = run_test()
report(duration)
log("INFO", f"Test completed in {duration}ms with {len(results)} operations")
# Allow time for span export before exit
log("INFO", "Waiting for spans to be exported...")
time.sleep(5)
if __name__ == "__main__":
main()
EOF
# Make the script executable
chmod +x traceloop_test.py
echo "Updated Python script traceloop_test.py created successfully!"
echo "Details:"
echo "- Date/time: 2025-10-17 11:14:46 (UTC)"
echo "- User: MphoMphego"
echo "- Service name: traceloop-py-MphoMphego"
echo ""
echo "To run the script:"
echo " pip install traceloop-sdk"
echo " ./traceloop_test.py"
Here are the outputs
Could you please update the script as shown below and then try running the application once?
os.environ["TRACELOOP_BASE_URL"] = "http://instana-agent.instana-agent.svc.cluster.local:4318" Traceloop.init(app_name="llm-tracing", telemetry_enabled=False)
I've previously made those changes to support disabling telemtry to POSTHOG. I did not get any traces in Instana with the script changes.
See: Resolution Applied -> https://github.com/traceloop/openllmetry/issues/3414#issuecomment-3456181314
I will test the script: https://github.com/traceloop/openllmetry/issues/3414#issuecomment-3460554367 and refer back
Thank you for your assistance thus far
@mmphego I also tested the script below and was able to get the traces. The change I made was updating the ENDPOINT to "http://instana-agent.instana-agent.svc.cluster.local:4318"
cat << 'EOF' > traceloop_test.py #!/usr/bin/env python3 # traceloop_test.py - Minimalist Python script for sending traces to Instana # Using Traceloop SDK for simplified GenAI observability import os import time import uuid from datetime import datetime # Import Traceloop components from traceloop.sdk import Traceloop from traceloop.sdk.decorators import workflow, task # Configuration USER_ID = "MphoMphego" SERVICE_NAME_VALUE = "traceloop-py-MphoMphego" ENDPOINT = "http://instana-agent.instana-agent.svc.cluster.local:4318" def log(level, message): """Simple logging function""" print(f"{datetime.now().strftime('%Y-%m-%d %H:%M:%S')} [{level}] - {message}") def setup_tracing(): """Set up Traceloop tracing with OTLP HTTP exporter for Instana""" log("INFO", f"Setting up Traceloop tracing for {SERVICE_NAME_VALUE}") # Initialize Traceloop with configuration Traceloop.init( app_name=SERVICE_NAME_VALUE, api_endpoint=ENDPOINT, disable_batch=False, # Use batching for better performance resource_attributes={ "user.id": USER_ID, } ) log("INFO", "Traceloop initialized successfully") @task(name="database.query") def database_query(): """Simulate database query operation""" log("INFO", "Executing database.query") time.sleep(0.2) return {"status": "success", "rows": 42} @task(name="llm.inference") def llm_inference(): """Simulate LLM inference operation""" log("INFO", "Executing llm.inference") time.sleep(0.2) return {"status": "success", "tokens": 150} @task(name="embedding.vector") def embedding_vector(): """Simulate embedding vector operation""" log("INFO", "Executing embedding.vector") time.sleep(0.2) return {"status": "success", "dimensions": 1536} @task(name="rag.retrieval") def rag_retrieval(): """Simulate RAG retrieval operation""" log("INFO", "Executing rag.retrieval") time.sleep(0.2) return {"status": "success", "documents": 5} @task(name="test.verification") def test_verification(): """Simulate test verification operation""" log("INFO", "Executing test.verification") time.sleep(0.2) return {"status": "success", "verified": True} @workflow(name="test_workflow") def run_test(): """Run simulated operations using Traceloop decorators""" trace_id = uuid.uuid4().hex log("INFO", f"Starting test with trace ID: {trace_id}") start_time = time.time() # Operations to simulate operations = [ "database.query", "llm.inference", "embedding.vector", "rag.retrieval", "test.verification" ] log("INFO", f"Simulating operations: {', '.join(operations)}") # Set workflow attributes using Traceloop's association Traceloop.set_association_properties({ "conversation_id": trace_id, "user_id": USER_ID }) # Run each operation results = [] log("INFO", "Simulating database.query") results.append(database_query()) time.sleep(0.1) log("INFO", "Simulating llm.inference") results.append(llm_inference()) time.sleep(0.1) log("INFO", "Simulating embedding.vector") results.append(embedding_vector()) time.sleep(0.1) log("INFO", "Simulating rag.retrieval") results.append(rag_retrieval()) time.sleep(0.1) log("INFO", "Simulating test.verification") results.append(test_verification()) time.sleep(0.1) duration = int((time.time() - start_time) * 1000) return duration, results def report(duration): """Generate a report of the test run""" print("\n==== INSTANA TRACE SUMMARY ====") print(f"Service: {SERVICE_NAME_VALUE}") print(f"User: {USER_ID}") print(f"Duration: {duration}ms") print("") print("Operations: workflow.execution, database.query, llm.inference, embedding.vector, rag.retrieval, test.verification") print(f"OTLP HTTP endpoint: {ENDPOINT}") print("") print(f"🔍 Search for: {SERVICE_NAME_VALUE} in Instana UI") print("==================================") def main(): """Main function to execute the trace test""" setup_tracing() duration, results = run_test() report(duration) log("INFO", f"Test completed in {duration}ms with {len(results)} operations") # Allow time for span export before exit log("INFO", "Waiting for spans to be exported...") time.sleep(5) if __name__ == "__main__": main() EOF # Make the script executable chmod +x traceloop_test.py echo "Updated Python script traceloop_test.py created successfully!" echo "Details:" echo "- Date/time: 2025-10-17 11:14:46 (UTC)" echo "- User: MphoMphego" echo "- Service name: traceloop-py-MphoMphego" echo "" echo "To run the script:" echo " pip install traceloop-sdk" echo " ./traceloop_test.py"Here are the outputs
Hi @adharshctr,
I ran the exact script in our environment, expecting traces to be exported to Instana, but unfortunately, I didn’t receive any.
Quick question: Is there a way to curl the endpoint to verify if a traceId was received—either using the Instana SDK or standard curl? Just looking for a way to confirm whether the trace is reaching Instana at all.
Hi @mmphego If you send a OpenTelemetry response body (with fixed traceId and spanId) to an endpoint, the receiver or exporter may overwrite those IDs with new ones because OpenTelemetry SDKs and collectors often generate fresh trace and span identifiers automatically to maintain trace integrity. This means your manually set IDs are ignored or replaced during serialization or ingestion, causing the call to appear with different traceId and spanId values.
You can check filtering it with Service Name, Call Name, Span attributes......
Filteration based on span attributes
Hi @mmphego Are you able to view the traces after changing the filter to Service Name, Call Name, or Span Attributes?
Hi @mmphego Are you able to view the traces after changing the filter to Service Name, Call Name, or Span Attributes?
🚀 🚀 To the moon!!!!
Managed to get traces into Instana using the test script.
Next, I’ll integrate Traceloop logic into our service to enable trace capture. I’ll keep you updated on the progress.
Question: What’s the current status of the Google ADK instrumentation (opentelemetry-instrumentation-google-adk)? Are there any blockers or updates I should be aware of?
Hi @mmphego Thank you. The Google ADK instrumentation has been prioritized, but development hasn’t started yet.