sentry-python icon indicating copy to clipboard operation
sentry-python copied to clipboard

Crash in thread sentry.profiler.ThreadScheduler on PyPy 3.10 / SDK 2.13.0

Open artempronevskiy opened this issue 1 year ago • 2 comments

How do you use Sentry?

Sentry Saas (sentry.io)

Version

2.13.0

Steps to Reproduce

  1. Crash happened on this line (see full traceback below)
  File "/home/artsiom/dev/project/statistics.project.com/.venv_pypy/lib/pypy3.10/site-packages/sentry_sdk/profiler/utils.py", line 172, in extract_stack
    f_back = raw_frame.f_back
  1. when I use the following decorator to wrap my function I want to analyze:
from functools import wraps
from typing import Any, Callable
 
import sentry_sdk
from sentry_sdk.tracing import Transaction
 
from configuration.utils import get_var_value_from_env
 
 
def get_sentry_transaction(func_name: str, transaction_name: str) -> Transaction:
    """
    Get the current transaction from the Sentry SDK or create a new one.
 
    @param func_name: name of the function
    @param transaction_name: name of the transaction
    @return: Sentry transaction
    """
    global_transaction = sentry_sdk.Hub.current.scope.transaction
 
    if global_transaction:
        transaction = global_transaction.start_child(op=func_name)
    else:
        transaction = sentry_sdk.start_transaction(op=func_name, name=transaction_name)
 
    return transaction
 
 
def sentry_instrument(
    transaction_name: str = "application",
    sentry_dsn_env_var_name: str = "SENTRY_DSN",
    traces_sample_rate: float = 1.0,
    profiles_sample_rate: float = 1.0,
    enable_tracing: bool = True,
    attach_stacktrace: bool = True,
    auto_enabling_integrations: bool = True,
    send_default_pii: bool = True,
    debug: bool = False,
) -> Callable[[Any], Any]:
    """
    Decorator to instrument the entrypoint function with Sentry SDK.
    It initializes the Sentry SDK with the provided DSN and other configurations.
    It also starts a transaction and a span for the entrypoint function.
    If the Sentry SDK is not initialized properly, it raises an AssertionError,
    as well as if the transaction or span is not started. All traces and metrics
    are sent to the Sentry server once the entrypoint function is executed and
    completed.
 
    @param transaction_name: name of the transaction
    @param sentry_dsn_env_var_name: name of the environment variable containing the Sentry DSN
    @param traces_sample_rate: sample rate for traces
    @param profiles_sample_rate: sample rate for profiles
    @param enable_tracing: flag to enable tracing
    @param attach_stacktrace: flag to attach stacktrace
    @param auto_enabling_integrations: flag to auto-enable integrations
    @param send_default_pii: flag to send default PII
    @param debug: flag to enable debug mode
    @return: decorated function
    """
 
    def decorator(func: Callable[[Any], Any]) -> Callable[[Any], Any]:
        @wraps(func)
        def wrapper(*args, **kwargs) -> Any:  # type: ignore[no-untyped-def]
            if not sentry_sdk.Hub.current.client:
                sentry_sdk.init(
                    dsn=get_var_value_from_env(sentry_dsn_env_var_name),
                    traces_sample_rate=traces_sample_rate,
                    profiles_sample_rate=profiles_sample_rate,
                    enable_tracing=enable_tracing,
                    attach_stacktrace=attach_stacktrace,
                    auto_enabling_integrations=auto_enabling_integrations,
                    send_default_pii=send_default_pii,
                    debug=debug,
                )
 
            if not sentry_sdk.is_initialized():
                raise AssertionError("Sentry client is not initialized properly")
 
            with (
                get_sentry_transaction(
                    func_name=func.__name__, transaction_name=transaction_name
                ),
                sentry_sdk.start_span(op=func.__name__),
            ):
                if not sentry_sdk.get_current_span():
                    raise AssertionError(
                        f"Sentry span is not started for {func.__name__}"
                    )
 
                try:
                    return func(*args, **kwargs)
                except Exception as e:
                    sentry_sdk.capture_exception(e)
                    raise e
 
        return wrapper
 
    return decorator
  1. pip freeze:
aiohappyeyeballs==2.3.7
aiohttp==3.10.4
aiosignal==1.3.1
annotated-types==0.7.0
async-timeout==4.0.3
attrs==24.2.0
boto3==1.35.0
boto3-stubs==1.35.0
botocore==1.35.0
botocore-stubs==1.35.0
certifi==2024.7.4
cffi==1.17.0.dev0
charset-normalizer==3.3.2
dnspython==2.6.1
email_validator==2.2.0
frozenlist==1.4.1
geoip2==4.8.0
greenlet==0.4.13
hpy==0.9.0
idna==3.7
jmespath==1.0.1
maxminddb==2.6.2
msgpack==1.0.8
multidict==6.0.5
mypy-boto3-s3==1.35.0
pydantic==2.8.2
pydantic_core==2.20.1
PyMySQL==1.1.1
python-dateutil==2.9.0.post0
pytz==2024.1
readline==6.2.4.1
requests==2.32.3
s3transfer==0.10.2
sentry-sdk==2.13.0
six==1.16.0
types-awscrt==0.21.2
types-s3transfer==0.10.1
typing_extensions==4.12.2
urllib3==2.2.2
yarl==1.9.4

Expected Result

No crash, all good.

Actual Result

Crash with traceback:

RPython traceback:
  File "rpython_jit_metainterp.c", line 22077, in BlackholeInterpreter__resume_mainloop
  File "rpython_jit_metainterp.c", line 48291, in BlackholeInterpreter_run
  File "rpython_jit_metainterp.c", line 48671, in BlackholeInterpreter_handle_exception_in_frame
Exception in thread sentry.profiler.ThreadScheduler:
 [sentry] DEBUG: Sending envelope [envelope with 1 items (error)] project:XXXXXXXXXXXXXXX host:YYYYYYYYYYYYYYYYYY.ingest.us.sentry.io
Traceback (most recent call last):
  File "/home/artsiom/Downloads/pypy3.10-v7.3.16-linux64/lib/pypy3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/home/artsiom/dev/project/statistics.project.com/.venv_pypy/lib/pypy3.10/site-packages/sentry_sdk/integrations/threading.py", line 99, in run
    return _run_old_run_func()
  File "/home/artsiom/dev/project/statistics.project.com/.venv_pypy/lib/pypy3.10/site-packages/sentry_sdk/integrations/threading.py", line 94, in _run_old_run_func
    reraise(*_capture_exception())
  File "/home/artsiom/dev/project/statistics.project.com/.venv_pypy/lib/pypy3.10/site-packages/sentry_sdk/utils.py", line 1659, in reraise
    raise value
  File "/home/artsiom/dev/project/statistics.project.com/.venv_pypy/lib/pypy3.10/site-packages/sentry_sdk/integrations/threading.py", line 92, in _run_old_run_func
    return old_run_func(self, *a, **kw)
  File "/home/artsiom/Downloads/pypy3.10-v7.3.16-linux64/lib/pypy3.10/threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
  File "/home/artsiom/dev/project/statistics.project.com/.venv_pypy/lib/pypy3.10/site-packages/sentry_sdk/profiler/transaction_profiler.py", line 732, in run
    self.sampler()
  File "/home/artsiom/dev/project/statistics.project.com/.venv_pypy/lib/pypy3.10/site-packages/sentry_sdk/profiler/transaction_profiler.py", line 617, in _sample_stack
    sample = [
  File "/home/artsiom/dev/project/statistics.project.com/.venv_pypy/lib/pypy3.10/site-packages/sentry_sdk/profiler/transaction_profiler.py", line 618, in <listcomp>
    (str(tid), extract_stack(frame, cache, cwd))
  File "/home/artsiom/dev/project/statistics.project.com/.venv_pypy/lib/pypy3.10/site-packages/sentry_sdk/profiler/utils.py", line 172, in extract_stack
    f_back = raw_frame.f_back
SystemError: unexpected internal exception (please report a bug): <InvalidVirtualRef object at 0x72edd9464fa8>; internal traceback was dumped to stderr

artempronevskiy avatar Sep 03 '24 16:09 artempronevskiy

If this is not related to Sentry SDK - I will open issue on PyPy issue tracker.

artempronevskiy avatar Sep 03 '24 16:09 artempronevskiy

Hey @artempronevskiy !

Does this also happen if your run it with CPython, or only in pypy? (We do not support pypy officially, so the code is not tested against pypy)

antonpirker avatar Sep 04 '24 12:09 antonpirker

Just saw that the original poster said it is not related to the Sentry SDK. Will close this.

antonpirker avatar Dec 10 '24 12:12 antonpirker