DataDog
[BUG]: Python int too large to convert to C long
Tracer Version(s)
doesnt work on:
- 2.20.0
- 2.19.3
- 2.17.5
- 2.15.4
- 2.14.7
- 2.13.2
- 2.12.4
- 2.11.8
- 2.10.7
Python Version(s)
Python 3.11
Pip Version(s)
pip 24.0
Bug Report
Our pytest job fail randomly.
Reproduction Code
No response
Error Logs
FAILED test/integration/mambo/status_banners/test_status_banners.py::test_banners__schedule_change_major[web-True] - RuntimeError: failed to pack span: <Span(id=4307602835879970050,trace_id=137759318570304310240605327865437761465,parent_id=17664599921369065393,name=flask.request)>. Exception: Python int too large to convert to C long
Libraries in Use
pytest==8.3.4 datadog==0.51.0 ddtrace==2.20.0
Operating System
Debian GNU/Linux 11 (bullseye)
Hey @lucas03,
Can you open a support ticket here: https://help.datadoghq.com/hc/en-us/requests/new.
The ddtrace-py library serializes trace data into msgpack. One of your spans likely contains an integer that is too large to encode as a C long (likely by this method).
To debug further you can configure a trace filter to capture span attributes (Span.duration_ns, Span.start_ns, Span._metrics, etc.) that contain integers that can not be converted to a C long. Once we identify the source of this value we can take steps to correct it.
Can you open a support ticket here
Your request (#2028629) has been received and will be reviewed by our team.
The ddtrace-py library serializes trace data into msgpack. One of your spans likely contains an integer that is too large to encode as a C long (likely by this method).
Could the problem be with datadog lib or something else? I have tried downgrading to the original version of ddtrace, but failing test is still there (flaky, not failing all the time)
To debug further you can configure a trace filter to capture span attributes (Span.duration_ns, Span.start_ns, Span._metrics, etc.) that contain integers that can not be converted to a C long. Once we identify the source of this value we can take steps to correct it.
Could you give me code snippet you have in mind? BTW I have ddtrace disabled in tests, but that doesn't seem to work.
requirements.txt
amqp==5.2.0
amqp==5.3.1
anyio==4.4.0
anyio==4.8.0
async-timeout==4.0.3
async-timeout==5.0.1
attrs==23.2.0
attrs==25.1.0
babel==2.15.0
babel==2.16.0
billiard==4.2.0
billiard==4.2.1
bleach==6.1.0
bleach==6.2.0
bottle==0.12.25
bottle==0.13.2
bytecode==0.15.1
bytecode==0.16.1
cachetools==5.4.0
cachetools==5.5.1
certifi==2024.7.4
certifi==2024.12.14
charset-normalizer==3.3.2
charset-normalizer==3.4.1
click==8.1.7
click==8.1.8
cryptography==43.0.3
cryptography==44.0.0
datadog==0.49.1
datadog==0.51.0
ddtrace==2.10.1
ddtrace==2.20.0
deprecated==1.2.14
deprecated==1.2.18
dnspython==2.6.1
dnspython==2.7.0
envier==0.5.2
envier==0.6.1
gevent==24.2.1
gevent==24.11.1
google-api-core==2.19.1
google-api-core==2.24.1
google-auth==2.32.0
google-auth==2.38.0
google-cloud-pubsub==2.23.0
google-cloud-pubsub==2.27.3
google-cloud-storage==2.18.0
google-cloud-storage==2.19.0
google-crc32c==1.5.0
google-crc32c==1.6.0
google-resumable-media==2.7.1
google-resumable-media==2.7.2
googleapis-common-protos==1.63.2
googleapis-common-protos==1.66.0
greenlet==3.0.3
greenlet==3.1.1
grpc-google-iam-v1==0.13.1
grpc-google-iam-v1==0.14.0
grpcio==1.65.1
grpcio==1.70.0
grpcio-status==1.65.1
grpcio-status==1.70.0
gunicorn==22.0.0
gunicorn==23.0.0
hiredis==3.0.0
hiredis==3.1.0
httpcore==1.0.5
httpcore==1.0.7
httpx==0.27.0
httpx==0.28.1
idna==3.7
idna==3.10
importlib-metadata==8.0.0
importlib-metadata==8.5.0
importlib-resources==6.4.0
importlib-resources==6.5.2
isodate==0.6.1
isodate==0.7.2
jinja2==3.1.4
jinja2==3.1.5
joserfc==1.0.0
joserfc==1.0.2
kiwi-cache==1.0.1
kiwi-cache==1.0.2
kiwi-google-metadata-auth==0.1.0
kiwi-google-metadata-auth==0.2.0
kiwi-json==0.10.0
kiwi-json==0.10.1
kiwi-structlog-config==0.3.3
kiwi-structlog-config==0.3.4
kombu==5.3.7
kombu==5.4.2
lxml==5.2.2
lxml==5.3.0
lz4==4.3.3
lz4==4.4.3
markupsafe==2.1.5
markupsafe==3.0.2
marshmallow==3.21.3
marshmallow==3.26.0
pathable==0.4.3
pathable==0.4.4
phonenumbers==8.13.40
phonenumbers==8.13.53
pillow==10.4.0
pillow==11.1.0
platformdirs==4.2.2
platformdirs==4.3.6
prompt-toolkit==3.0.47
prompt-toolkit==3.0.50
proto-plus==1.24.0
proto-plus==1.26.0
protobuf==5.27.2
protobuf==5.29.3
psycopg==3.1.19
psycopg==3.2.4
psycopg-binary==3.1.19
psycopg-binary==3.2.4
pyasn1==0.6.0
pyasn1==0.6.1
pyasn1-modules==0.4.0
pyasn1-modules==0.4.1
pydantic==2.8.2
pydantic==2.10.6
pydantic-core==2.20.1
pydantic-core==2.27.2
pyjwt==2.8.0
pyjwt==2.10.1
pyparsing==3.1.2
pyparsing==3.2.1
pytz==2024.1
pytz==2024.2
pyyaml==6.0.1
pyyaml==6.0.2
redis==5.0.7
redis==5.2.1
request-session==0.15.0
request-session==0.16.2
ruamel-yaml==0.18.6
ruamel-yaml==0.18.10
ruamel-yaml-clib==0.2.8
ruamel-yaml-clib==0.2.12
schwifty==2024.6.1
schwifty==2025.1.0
sentry-sdk==2.11.0
sentry-sdk==2.12.0
simplejson==3.19.2
simplejson==3.19.3
six==1.16.0
six==1.17.0
sqlalchemy==2.0.31
sqlalchemy==2.0.37
structlog==24.4.0
structlog==25.1.0
types-python-dateutil==2.9.0.20240316
types-python-dateutil==2.9.0.20241206
tzdata==2024.1
tzdata==2025.1
urllib3==2.2.2
urllib3==2.3.0
webob==1.8.7
webob==1.8.9
wrapt==1.16.0
wrapt==1.17.2
xlsxwriter==3.2.0
xlsxwriter==3.2.1
xmltodict==0.13.0
xmltodict==0.14.2
zeep==4.2.1
zeep==4.3.1
zipp==3.19.2
zipp==3.21.0
zope-interface==6.4.post2
zope-interface==7.2
and test-requirements.txt
alembic==1.13.2
alembic==1.14.1
certifi==2024.7.4
certifi==2024.12.14
charset-normalizer==3.3.2
charset-normalizer==3.4.1
coverage==7.6.0
coverage==7.6.10
cryptography==43.0.3
cryptography==44.0.0
faker==35.0.0
faker==26.0.0
fakeredis==2.23.3
fakeredis==2.26.2
idna==3.7
idna==3.10
kiwi-json==0.10.0
kiwi-json==0.10.1
mako==1.3.5
mako==1.3.8
markupsafe==2.1.5
markupsafe==3.0.2
mypy==1.11.0
mypy==1.14.1
polyfactory==2.16.2
polyfactory==2.18.1
pyaml==24.7.0
pyaml==25.1.0
pyparsing==3.1.2
pyparsing==3.2.1
pytest==8.3.2
pytest==8.3.4
pytest-cov==5.0.0
pytest-cov==6.0.0
pytest-randomly==3.15.0
pytest-randomly==3.16.0
pyyaml==6.0.1
pyyaml==6.0.2
redis==5.0.7
redis==5.2.1
six==1.16.0
six==1.17.0
soupsieve==2.5
soupsieve==2.6
sql-metadata==2.12.0
sql-metadata==2.15.0
sqlalchemy==2.0.31
sqlalchemy==2.0.37
sqlparse==0.5.1
sqlparse==0.5.3
types-bleach==6.1.0.20240331
types-bleach==6.2.0.20241123
types-cachetools==5.4.0.20240717
types-cachetools==5.5.0.20240820
types-cffi==1.16.0.20240331
types-cffi==1.16.0.20241221
types-html5lib==1.1.11.20240228
types-html5lib==1.1.11.20241018
types-python-dateutil==2.9.0.20240316
types-python-dateutil==2.9.0.20241206
types-pytz==2024.1.0.20240417
types-pytz==2024.2.0.20241221
types-pyyaml==6.0.12.20240724
types-pyyaml==6.0.12.20241230
types-redis==4.6.0.20240726
types-redis==4.6.0.20241004
types-requests==2.32.0.20240712
types-requests==2.32.0.20241016
types-setuptools==71.1.0.20240726
types-setuptools==75.8.0.20250110
types-simplejson==3.19.0.20240310
types-simplejson==3.19.0.20241221
urllib3==2.2.2
urllib3==2.3.0
waitress==3.0.0
waitress==3.0.2
webob==1.8.7
webob==1.8.9
webtest==3.0.0
webtest==3.0.3
this is the traceback from tests:
_________________________ test_pending_flight_changes __________________________
self = <_TracedIterable at 0x7c1c724aee80 for ClosingIterator at 0x7c1c72fc23d0>
def __next__(self):
try:
if self._self_wrapped_is_iterator:
return next(self._wrapped_iterator)
else:
> return next(self.__wrapped__)
/usr/local/lib/python3.11/site-packages/ddtrace/_trace/trace_handlers.py:61:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <werkzeug.wsgi.ClosingIterator object at 0x7c1c72fc23d0>
def __next__(self) -> bytes:
> return self._next()
E StopIteration
/usr/local/lib/python3.11/site-packages/werkzeug/wsgi.py:500: StopIteration
During handling of the above exception, another exception occurred:
> ???
ddtrace/internal/_encoding.pyx:501:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> ???
E OverflowError: Python int too large to convert to C long
ddtrace/internal/_encoding.pyx:892: OverflowError
During handling of the above exception, another exception occurred:
app = <test.integration.mambo.conftest.MamboClient object at 0x7c1c72e3f1d0>
scenario_return_trip = Scenario(booking=<kw.naomi.booking.bookings.Booking object at 0x7c1c7290e250>, user=<kw.naomi.booking.users.User objec...cGxlLm9yZyIsImJyYW5kIjoia2l3aWNvbSIsImV4cCI6MTczOTMwNDM5Mn0.J7VAvQgWnYWMV_9HWHmbZDaJq_Igx5Vy2bdRU2TlP6s'}, payments=[])
mock_flight_changes = (<kw.naomi.booking.alternatives.PendingFlightChange object at 0x7c1c859d20d0>, <kw.naomi.booking.alternatives.PendingFlightChange object at 0x7c1c724ad250>)
def test_pending_flight_changes(app, scenario_return_trip, mock_flight_changes):
response = app.get(f"/bookings/{scenario_return_trip.booking.id}/itinerary_changes/pending")
assert response.status_code == 200
> assert response.json == {
"pending_flight_changes": [{"id": mock_flight_changes[1].id}, {"id": mock_flight_changes[0].id}]
}
test/integration/mambo/itinerary_changes/test_pending_flight_changes.py:89:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/local/lib/python3.11/site-packages/werkzeug/wrappers/response.py:646: in json
return self.get_json()
/usr/local/lib/python3.11/site-packages/werkzeug/wrappers/response.py:672: in get_json
data = self.get_data()
/usr/local/lib/python3.11/site-packages/werkzeug/wrappers/response.py:314: in get_data
self._ensure_sequence()
/usr/local/lib/python3.11/site-packages/werkzeug/wrappers/response.py:376: in _ensure_sequence
self.make_sequence()
/usr/local/lib/python3.11/site-packages/werkzeug/wrappers/response.py:391: in make_sequence
self.response = list(self.iter_encoded())
/usr/local/lib/python3.11/site-packages/werkzeug/wrappers/response.py:50: in _iter_encoded
for item in iterable:
/usr/local/lib/python3.11/site-packages/werkzeug/wsgi.py:500: in __next__
return self._next()
/usr/local/lib/python3.11/site-packages/ddtrace/_trace/trace_handlers.py:63: in __next__
self._finish_spans()
/usr/local/lib/python3.11/site-packages/ddtrace/_trace/trace_handlers.py:80: in _finish_spans
self._self_span.finish()
/usr/local/lib/python3.11/site-packages/ddtrace/_trace/span.py:313: in finish
self._finish_ns(time_ns())
/usr/local/lib/python3.11/site-packages/ddtrace/_trace/span.py:326: in _finish_ns
cb(self)
/usr/local/lib/python3.11/site-packages/ddtrace/_trace/tracer.py:869: in _on_span_finish
p.on_span_finish(span)
/usr/local/lib/python3.11/site-packages/ddtrace/_trace/processor/__init__.py:370: in on_span_finish
self._writer.write(spans)
/usr/local/lib/python3.11/site-packages/ddtrace/internal/writer/writer.py:324: in write
self._write_with_client(client, spans=spans)
/usr/local/lib/python3.11/site-packages/ddtrace/internal/writer/writer.py:347: in _write_with_client
client.encoder.put(spans)
ddtrace/internal/_encoding.pyx:844: in ddtrace.internal._encoding.MsgpackEncoderV05.put
???
ddtrace/internal/_encoding.pyx:845: in ddtrace.internal._encoding.MsgpackEncoderV05.put
???
ddtrace/internal/_encoding.pyx:851: in ddtrace.internal._encoding.MsgpackEncoderV05.put
???
ddtrace/internal/_encoding.pyx:848: in ddtrace.internal._encoding.MsgpackEncoderV05.put
???
ddtrace/internal/_encoding.pyx:511: in ddtrace.internal._encoding.MsgpackEncoderBase.put
???
ddtrace/internal/_encoding.pyx:515: in ddtrace.internal._encoding.MsgpackEncoderBase.put
???
ddtrace/internal/_encoding.pyx:538: in ddtrace.internal._encoding.MsgpackEncoderBase.put
???
ddtrace/internal/_encoding.pyx:519: in ddtrace.internal._encoding.MsgpackEncoderBase.put
???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> ???
E RuntimeError: failed to pack span: <Span(id=9557175821412020941,trace_id=137800164174119564133353386103302162856,parent_id=17168043924156404373,name=flask.request)>. Exception: Python int too large to convert to C long
ddtrace/internal/_encoding.pyx:503: RuntimeError
I found how I introduced this issue. As a part of requirements upgrade, I squashed a small change in:
from this:
from ddtrace import config, tracer
from ddtrace.sampler import RateSampler
tracer.configure(enabled=False)
to this:
if not test:
from ddtrace import tracer
tracer.configure(enabled=False)
so this no longer runs in our pytest jobs. This causes the span to be constructed with that big trace_id.
I don't know why, as I'd expect spans not to be created if not explicitly configured. And we have this in pytest-ini file:
addopts = --strict-markers -p no:ddtrace -p no:ddtrace.pytest_bdd -p no:ddtrace.pytest_benchmark -p no:ddtrace.internal._unpatched
but anyway, fixed this issue by adding this to our conftest.py file:
tracer.configure(enabled=False)
Hello @lucas03,
Thank you for your update. We're pleased to hear that your issue has been resolved. I will close this ticket, but feel free to reopen it if the problem reoccurs.
Thank you again!
Hey, I am trying to upgrade ddtrace now, from 2.21.0 to 3.9.0. However I see a flaky test again:
FAILED test/integration/mambo/shopping_basket/handlers/test_insurances.py::test_hours_to_first_departure_validation - RuntimeError: failed to pack span: <Span(id=7263596379739779291,trace_id=138606696489625485713255291423573495442,parent_id=2152773425742347048,name=flask.request)>. Exception: Python int too large to convert to C long
This is main part of diff:
but I don't have permission to reopen issue, @rachelyangdog can you?
@emmettbutler sorry to tag you here, I've seen you had last commit. How do I reopen this, or should I create a new issue? Error message seems the same.
Hi @lucas03 ,
We saw that you previously opened a support ticket that you marked as solved. Could you please open a new support ticket here: https://help.datadoghq.com/hc/en-us/requests/new and also provide the following information in the support ticket:
- Enable tracer debug logs with
DD_TRACE_DEBUG=trueand output (more instructions here).
This will allow us to troubleshoot further, and we can help you through the support channel as it is more secure (that way there is no need to reply on Github).
