[BUG] Cannot connect to telemetry.crewai.com

[slimshreydy]

Description Cannot connect to telemetry.crewai.com.

Steps to Reproduce Provide a step-by-step process to reproduce the behavior: Make any stub code for a crew completing a task. Your console will be flooded with errors saying it can't connect to telemetry.crewai.com. It seems like this error logging should be suppressed as its not essential to the program, and it obfuscates the actual valuable console output.

Expected behavior No logging when there's a failure to connect to the (optional) telemetry server. My code and deployments should not suffer because the crewai.com site is down.

Screenshots/Code snippets Here's the simple code I'm running:

from crewai import Crew, Task, Agent
from crewai_tools import tool

@tool("Add tool")
def add(a: int, b: int) -> int:
    """
    Adds two numbers together

    :param a: First number to add
    :param b: Second number to add
    :return: The sum of the two numbers `a` and `b`.
    """
    return a + b
math_agent = Agent(
    role="Math",
    goal="Add numbers",
    backstory="I am an agent that can add 2 numbers together.",
    tools=[add],
    allow_delegation=False,
)
add_task = Task(
    description="Add numbers {a} and {b}",
    expected_output="The sum of 3 and 5 is 8.",
    agent=math_agent,
)
crew = Crew(agents=[math_agent], tasks=[add_task], max_rpm=100)
result = crew.kickoff(inputs={ "a": 10, "b": 20 })

It seems the error may be occurring because crewai.com is down. When I access crewai.com, it redirects to some firewall from my ISP saying crewai.com's certificate is invalid.

Environment Details:

• Operating System: macOS Sonoma
• Python Version: 3.11
• crewAI Version: 0.41.1
• crewAI Tools Version: 0.4.26

Logs

Here's what gets printed in my console when I run the stub code above.

It does print the correct answer of 30 (albeit not with the correct format...) but then it dumps a bunch of errors involving failed connections to the telemetry server.

30

2024-08-10 23:57:47,791 - 11391905792 - __init__.py-__init__:369 - ERROR: Exception while exporting Span batch.
Traceback (most recent call last):
  File "/Users/shrey/code/lambdas/venv/lib/python3.11/site-packages/urllib3/connection.py", line 198, in _new_conn
    sock = connection.create_connection(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/shrey/code/lambdas/venv/lib/python3.11/site-packages/urllib3/util/connection.py", line 85, in create_connection
    raise err
  File "/Users/shrey/code/lambdas/venv/lib/python3.11/site-packages/urllib3/util/connection.py", line 73, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 61] Connection refused

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/shrey/code/lambdas/venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 793, in urlopen
    response = self._make_request(
               ^^^^^^^^^^^^^^^^^^^
  File "/Users/shrey/code/lambdas/venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 491, in _make_request
    raise new_e
  File "/Users/shrey/code/lambdas/venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 467, in _make_request
    self._validate_conn(conn)
  File "/Users/shrey/code/lambdas/venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 1099, in _validate_conn
    conn.connect()
  File "/Users/shrey/code/lambdas/venv/lib/python3.11/site-packages/urllib3/connection.py", line 616, in connect
    self.sock = sock = self._new_conn()
                       ^^^^^^^^^^^^^^^^
  File "/Users/shrey/code/lambdas/venv/lib/python3.11/site-packages/urllib3/connection.py", line 213, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x2a24c0590>: Failed to establish a new connection: [Errno 61] Connection refused

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/shrey/code/lambdas/venv/lib/python3.11/site-packages/requests/adapters.py", line 486, in send
    resp = conn.urlopen(
           ^^^^^^^^^^^^^
  File "/Users/shrey/code/lambdas/venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 847, in urlopen
    retries = retries.increment(
              ^^^^^^^^^^^^^^^^^^
  File "/Users/shrey/code/lambdas/venv/lib/python3.11/site-packages/urllib3/util/retry.py", line 515, in increment
    raise MaxRetryError(_pool, url, reason) from reason  # type: ignore[arg-type]
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='telemetry.crewai.com', port=4319): Max retries exceeded with url: /v1/traces (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x2a24c0590>: Failed to establish a new connection: [Errno 61] Connection refused'))

(It'll keep printing this for each retry and on each action that triggers CrewAI telemetry -- I've omitted the excess logs here.)

Possible Solution Suppress logging output when the telemetry server connections fail. Telemetry is nonessential for end users of crewai libraries; it's understandable to have invisible telemetry to improve the library, but it's not good to have it hamper the developer experience here (or, for anyone using crewai in production, blow up someone's cloud logging bill and maybe even their Slack/PagerDuty).

Additional context Add any other context about the problem here.

[LiAnQing279]

When I was debugging, a lot of debugging information https://telemetry.crewai.com/ could not be accessed, which seriously interfered with the normal debugging process.

[LiAnQing279]

I've seen previous discussions that there are two ways to turn this off.

From what I understand, telemetry is currently hardwired into the system. If this is incorrect, please let me know.

You are correct, but you can (or at least could) disable telemetry by adding the following code snippet:

from crewai.telemetry import Telemetry

def noop(*args, **kwargs):
print("Telemetry method called and noop'd\n")
pass

for attr in dir(Telemetry):
if callable(getattr(Telemetry, attr)) and not attr.startswith("__"):
setattr(Telemetry, attr, noop)

Now add the environment variable at the top of the script:

import os
from crewai import Agent, Task, Crew, Process
from crewai_tools import SerperDevTool

os.environ["OTEL_SDK_DISABLED"] = "true"

[slimshreydy]

Appreciate it @LiAnQing279, will definitely start applying this to my CrewAI runs.

Would also appreciate a response from a CrewAI maintainer, since I imagine devs being forced to gut their telemetry thru introspection just to get the library to work is not the intended devX

[slimshreydy]

@joaomdmoura any thoughts on this? The crewai.com site is having certificate issues yet again, and once again anyone who's attempting to develop with CrewAI right now is likely experiencing a ridiculous amount of extraneous log output.

[theCyberTech]

develop

What certificate issues are you referring to? Certificate is not expired and is provided by a well known issuer

[slimshreydy]

Have been trying to access crewai.com and it's been occasionally getting blocked by my ISP. They're claiming it's because the certificate is invalid.

It could be an issue with my ISP's firewall or something, but even if it's not a crewai.com issue, the point stands -- if a connection to telemetry.crewai.com fails for any reason whatsoever (whether a local issue, firewall issue, or an error on CrewAI's end), it shouldn't spam a whole wall of errors in the logging output. Telemetry exists only for the benefit of the CrewAI team, so my thought is that developers using this library shouldn't see any stdout or stderr logging output re: telemetry (unless explicitly enabled).

[slimshreydy]

For what it's worth I just tried accessing crewai.com in my browser and I got this:

Not sure if there's just something wonky going on with my machine/network, but I don't really have any special configs I'm aware of. So even if this isn't affecting everyone, it's likely affecting more people than just me...

[theCyberTech]

Most likely your browser does not have the CA cert for that certificate, make sure your device is completely up to dat

[theCyberTech]

Also that message does not come from your isp, that's a browser message

[slimshreydy]

@theCyberTech I've gotten messages from both my browser and my ISP in the past re: this issue (see the original post; my ISP CenturyLink was blocking access at one point too)

Also the issue only pops up occasionally; doesn't seem like a persistent issue, which leads me to believe the problem is not my laptop or networking setup.

Regardless, the fundamental issue remains; regardless of what the issue is, telemetry should never log errors to the console (unless someone uses verbose logging), since developers don't care if telemetry is failing. If telemetry fails, it should fail silently. Otherwise, the best approach for devs here is to just block telemetry completely.

[github-actions[bot]]

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[github-actions[bot]]

This issue was closed because it has been stalled for 5 days with no activity.

[urlan]

Months later and this error continues:

requests.exceptions.SSLError: HTTPSConnectionPool(host='telemetry.crewai.com', port=4319): Max retries exceeded with url: /v1/traces (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1000)')))

[Nirupam-Naragund]

The error is back. Anyone has any idea on how to fix?

[XinyueZ]

The error is back. Anyone has any idea on how to fix?

os.environ["OTEL_SDK_DISABLED"] = "true"

[dmwheel1]

The easiest way to disable telemetry is to add the following to your .env file in your project CREWAI_DISABLE_TELEMETRY=true

[FaresElmenshawii]

I’m seeing the same error. I stopped it by deleting the related code in the source, since setting environment variables didn’t work.

This is, at best, deceptive behavior enabling telemetry without any consent or warning. Many production environments don’t allow outbound connections, so this kind of default behavior can cause serious issues.