pydevd error when debugging with wrapt

[endmarsfr]

Hi there,

I tested with python 3.12 & python 3.12.1 (x64) on windows 10 22H2. When I debug with pycharm 2023.3 and wrapt-1.16.0, I get this error:

File "C:\Users\endmarsfr\AppData\Local\Programs\Python\Lib\site-packages\wrapt\decorators.py", line 239, in _build return AdapterWrapper(wrapped=wrapped, wrapper=wrapper, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "_pydevd_bundle/pydevd_pep_669_tracing_cython.pyx", line 504, in _pydevd_bundle.pydevd_pep_669_tracing_cython.PyRaiseCallback.__call__ frame = self.frame File "_pydevd_bundle/pydevd_pep_669_tracing_cython.pyx", line 47, in _pydevd_bundle.pydevd_pep_669_tracing_cython.PEP669CallbackBase.frame while frame and isinstance(frame.f_locals.get('self'), PEP669CallbackBase): ValueError: wrapper has not been initialized python-BaseException

Kind regards

[GrahamDumpleton]

The debugger is trying to access attributes of the wrapt ProxyObject instance before it is initialized properly. The exception is due to that access when it is in an uninitialized state.

All you can really do is step over the initialization of the decorator and not into it, or see if it you can run to exit of the function in hope that the debugger isn't still trying to introspect the object instance while it is in the uninitialized state.

[endmarsfr]

Hi Graham,

Do you think the problem is with the debugger? All my scripts that use wrapt can no longer be debugged. The debugger seems to work fine with other scripts.

kind regards

[GrahamDumpleton]

It is the nature of debuggers that they may provide a live view of objects on the stack or otherwise somehow in scope. If that feature is enabled, possibly simply by having a tracing view in the debugger visible, it may try to aggressively introspect objects, which could be an issue when objects can be in intermediate initialised states. That said, it should ignore any exceptions when doing introspection and not fail, so am not sure, especially not on the limited information you have given.

If you could record a video and post it on YouTube or somewhere else where I could watch how you use the debugger and what you are doing, plus where the error shows, that would be helpful as I don't use Python debuggers myself so don't know what they might actually be doing.

[cerrussell]

Hi @GrahamDumpleton, I'm having the same issue. Here's a video and traceback.

Additional details:

• I am working on the feature/csaf-refactor branch of owasp-dep-scan/dep-scan.
• I wanted to examine the output of the variables returned by the summarise function of line 1073 of cli.py and set my breakpoint a few lines later.
• Our pygithub dependency uses wrapt, and the import of depscan.lib.github into cli.py means this happens whenever I try to debug.

[GrahamDumpleton]

Can you confirm that if you tell the debugger to continue from that point that it then will run until the next break point or exception?

Also, is there a way to tell the debugger to ignore exceptions which occur at certain points in the code when it is doing its introspection and continue running anyway?

As alluded to before, the issue is that the debugger is inserting a tracing function which then tries to introspect objects for every Python opcode (??) execution. This means it could easily trigger exceptions when objects are in an intermediary initialised state.

Right now am not sure what options I have to avoid this occurring. One might be to raise an AttributeError instead of ValueError exception. The AttributeError is often treated differently by things and causes stuff to fallback to some other action and do something differently or just ignore things.

If you wanted to see if changing the exception type would help, you would need to get down a copy of wrapt source code and change all places which raise ValueError with message "wrapper has not been initialized" to AttributeError instead. Install from that modified source code into your local virtual environment for testing and try again.

[cerrussell]

@GrahamDumpleton

Can you confirm that if you tell the debugger to continue from that point that it then will run until the next break point or exception?

If I tell it to continue, the debugger just terminates with that traceback in the console (plus "Process finished with exit code 1 at the bottom). There is no way of continuing to debug.

I'll give changing the exception type a try.

[hartym]

I do see the same problem, that I could reduce to the following minimal case:

import wrapt

@wrapt.decorator
def foo(wrapped, instance, args, kwargs):
    pass

Running this (useless) code works (as in, it does nothing but does not raise anything) but if I run it through the intellij/pycharm debugger, the same exception as seen by the OP stops the process (no resume if I ask the debugger to continue).

Here is the stack trace and context (paths shortened):

.../bin/python3.12 -X pycache_prefix=~/Library/Caches/JetBrains/IntelliJIdea2023.3/cpython-cache ~/Library/Application Support/JetBrains/IntelliJIdea2023.3/plugins/python/helpers/pydev/pydevd.py --multiprocess --qt-support=auto --client 127.0.0.1 --port 49284 --file test_github.py 
Connected to pydev debugger (build 233.13135.103)
Traceback (most recent call last):
  File "~/Library/Application Support/JetBrains/IntelliJIdea2023.3/plugins/python/helpers/pydev/pydevd.py", line 1527, in _exec
    pydev_imports.execfile(file, globals, locals)  # execute the script
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/Library/Application Support/JetBrains/IntelliJIdea2023.3/plugins/python/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "test_github.py", line 4, in <module>
    @wrapt.decorator
     ^^^^^^^^^^^^^^^
  File "lib/python3.12/site-packages/wrapt/decorators.py", line 427, in decorator
    return _build(wrapper, _wrapper, adapter=decorator)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "lib/python3.12/site-packages/wrapt/decorators.py", line 239, in _build
    return AdapterWrapper(wrapped=wrapped, wrapper=wrapper,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "_pydevd_bundle/pydevd_pep_669_tracing_cython.pyx", line 504, in _pydevd_bundle.pydevd_pep_669_tracing_cython.PyRaiseCallback.__call__
  File "_pydevd_bundle/pydevd_pep_669_tracing_cython.pyx", line 47, in _pydevd_bundle.pydevd_pep_669_tracing_cython.PEP669CallbackBase.frame
ValueError: wrapper has not been initialized
python-BaseException

Process finished with exit code 1

The "solution" I have for now is to go back to python3.11, the issue does not exist with it.

Any help will be greatly appreciated. As it is, it makes anything using wrapt unusable with jetbrains' debugger (like in my case, PyGithub or deprecated.deprecated)

[GrahamDumpleton]

Do you get the same problem if you set and export the environment variable:

WRAPT_DISABLE_EXTENSIONS=true

This will clarify whether it is only specific to C extension variant of wrapt, or also the pure Python version.

If know the pure Python version is affected, then I can try and create a small test case (independent of wrapt), which I believe may replicate the specific code I think is giving the debugger a problem.

If that shows the issue, then the exception type change as already explained above can be tried.

If can narrow it done that way, then can provide the details to debugger authors and they can look at the issue of why they fail on the particular code.

[cerrussell]

@GrahamDumpleton I get the same problem even with that environment variable set.

This problem could be related to implementation of PEP 669 in Python 3.12. I meant to check this, but it slipped my mind until I saw @hartym's comment.

I can confirm that I do not have this problem in a Python 3.10 or 3.11 virtual environment. Looking at the pydev issue tracker, I don't see anything about this issue definitively, but there are a couple of other issues occurring in 3.12 that may have the same origin, including that pydev's own tests are failing.

@hartym May I ask what OS you're using? I can see that the original OP is using Windows, as am I. Given the tests were failing for Windows and Mac in the last pydev release, but not Linux, I'm wondering if this problem is OS-specific.

[hartym]

@cerrussell I'm using MacOS 12.6

@GrahamDumpleton I tried running again my "test case" using python 3.12 and with WRAPT_DISABLE_EXTENSIONS=true, unfortunately, the behaviour is quite similar to before, the only difference is that now the debugger "understands" what file/line matches the error, which was not the case with the cython frames.

Without extensions

With extensions

[cerrussell]

@hartym @endmarsfr @GrahamDumpleton Found a workaround on the PyCharm issues tracker. In PyCharm, go to Help > Find Action > Registry and uncheck the box for python.debug.low.impact.monitoring.api.

~~Debugger did not terminate but am still trying to get the variables to show (this may be from me tinkering with other settings trying to find a solution).~~ Works fine now.

[endmarsfr]

Hi @cerrussell Thanks for the workaround. I wish you happy holiday celebrations

[GrahamDumpleton]

Can someone try and run this little test program through the debugger to see if it fails in same was as debugger has so far.

class Object: pass

class Wrapper:

    def __init__(self, force_error=False):
        if force_error:
            print(self.__wrapped__)
        self.func()
        object.__setattr__(self, "__wrapped__", Object())

    @property
    def __dict__(self):
        return self.__wrapped__.__dict__

    def __getattr__(self, name):
        if name == '__wrapped__':
            raise ValueError('wrapper has not been initialised')

        return getattr(self.__wrapped__, name)

    def __setattr__(self, name, value):
        setattr(self.__wrapped__, name, value)

    def func(self):
        pass


wrapper = Wrapper()

wrapper.xxx = True
print(wrapper.xxx)

print(wrapper.__dict__)
print(wrapper.__wrapped__.__dict__)

# wrapper = Wrapper(force_error=True)

If it doesn't I will try and tweak the example further as right now it doesn't mirror exactly what pure Python version of wrapt code does.

[cerrussell]

@GrahamDumpleton It didn't fail for me.

[GrahamDumpleton]

I just modified it so there would be code run before the setattr. Can you try again if you use the original version.

[GrahamDumpleton]

I have made another change to the code to try. This time added a __dict__ property. The debugger is likely using dir() to introspect the object but that fails. The prior screen shots since they don't expand all the stack frames, or can't do so since debugger is itself C code, you can't verify what it fails on.

Fudging up a simple debugger which tries to introspect an object, one would see:

started Object
started Wrapper
started caller
  caller: 49
started __init__
Traceback (most recent call last):
  File "/private/tmp/debug.py", line 55, in <module>
started __init__
started __init__
started getstate
started decode
    caller()
  File "/private/tmp/debug.py", line 49, in caller
started __init__
started __init__
started getstate
started decode
    wrapper = Wrapper()
              ^^^^^^^^^
  File "/private/tmp/debug.py", line 8, in __init__
started __init__
started __init__
started getstate
started decode
    def __init__(self, force_error=False):

  File "/private/tmp/debugger.py", line 51, in start_handler
started __init__
started __init__
started getstate
started decode
    dir(frame.f_locals.get('self'))
  File "/private/tmp/debug.py", line 23, in __dict__
started __init__
started __init__
started getstate
started decode
    return self.__wrapped__.__dict__
           ^^^^^^^^^^^^^^^^
  File "/private/tmp/debug.py", line 27, in __getattr__
started __init__
started __init__
started getstate
started decode
    raise ValueError('wrapper has not been initialised')
ValueError: wrapper has not been initialised

So try with latest code from comment https://github.com/GrahamDumpleton/wrapt/issues/257#issuecomment-1872599518

[cerrussell]

@GrahamDumpleton No errors still.

[GrahamDumpleton]

Try again with this one. We just need to try and work out which special method might be tripping it up.

class Object: pass

class Wrapper:

    def __init__(self, force_error=False):
        if force_error:
            print(self.__wrapped__)
        self.func()
        object.__setattr__(self, "__wrapped__", Object())

    @property
    def __dict__(self):
        return self.__wrapped__.__dict__

    def __getattr__(self, name):
        if name == '__wrapped__':
            raise ValueError('wrapper has not been initialised')

        return getattr(self.__wrapped__, name)

    def __setattr__(self, name, value):
        setattr(self.__wrapped__, name, value)

    @property
    def __name__(self):
        return self.__wrapped__.__name__

    @__name__.setter
    def __name__(self, value):
        self.__wrapped__.__name__ = value

    @property
    def __class__(self):
        return self.__wrapped__.__class__

    @__class__.setter
    def __class__(self, value):
        self.__wrapped__.__class__ = value

    def __dir__(self):
        return dir(self.__wrapped__)

    def __str__(self):
        return str(self.__wrapped__)

    def __bytes__(self):
        return bytes(self.__wrapped__)

    def __repr__(self):
        return '<{} at 0x{:x} for {} at 0x{:x}>'.format(
                type(self).__name__, id(self),
                type(self.__wrapped__).__name__,
                id(self.__wrapped__))

    def func(self):
        pass


wrapper = Wrapper()

wrapper.xxx = True
print(wrapper.xxx)

print(wrapper.__dict__)
print(wrapper.__wrapped__.__dict__)

# wrapper = Wrapper(force_error=True)

[cerrussell]

@GrahamDumpleton That one got the error!

[GrahamDumpleton]

Do you get a nice stack traceback so we know which access is the problem?

[cerrussell]

Here ya go. traceback.txt

I think the repr exception is from PyCharm trying to render the variables after the initial exception.

[zyoung-rc]

Seems to be the __class__ property in your example. Here is the example in it's minimally failing form

class Object: pass

class Wrapper:

    def __init__(self, force_error=False):
        if force_error:
            print(self.__wrapped__)
        self.func()
        object.__setattr__(self, "__wrapped__", Object())

    def __getattr__(self, name):
        if name == '__wrapped__':
            raise ValueError('wrapper has not been initialised')

        return getattr(self.__wrapped__, name)

    @property
    def __class__(self):
        return self.__wrapped__.__class__

    @__class__.setter
    def __class__(self, value):
        self.__wrapped__.__class__ = value

    def func(self):
        pass


wrapper = Wrapper()

[zyoung-rc]

I use pytest and this fixture temporarily fixes the problem until a permanent fix can be made. It can easily be adapted to unittest or another framework.

@pytest.fixture(scope="session")
def patch_wrapt_for_pycharm():
    from wrapt import decorators, FunctionWrapper
    from wrapt.decorators import AdapterWrapper, _AdapterFunctionSurrogate

    class _PatchedAdapterFunctionSurrogate(_AdapterFunctionSurrogate):
        @property
        def __class__(self):
            try:
                return super().__class__
            except ValueError:
                return type(self)


    class PatchedAdapterWrapper(AdapterWrapper):
        def __init__(self, *args, **kwargs):
            adapter = kwargs.pop("adapter")
            FunctionWrapper.__init__(self, *args, **kwargs)
            self._self_surrogate = _PatchedAdapterFunctionSurrogate(self.__wrapped__, adapter)
            self._self_adapter = adapter

        @property
        def __class__(self):
            try:
                return super().__class__
            except ValueError:
                return type(self)

    with pytest.MonkeyPatch.context() as patch:
        patch.setattr(decorators, "AdapterWrapper", PatchedAdapterWrapper)
        yield

It's important to note this patching needs to run before @wrapt.decorator is used so you may need to play with the placement.

This test will not raise a PyCharm debugging error

def test_wrapt(patch_wrapt_for_pycharm):
    @wrapt.decorator
    def pass_through(wrapped, instance, args, kwargs):
        return wrapped(*args, **kwargs)

    @pass_through
    def function():
        print("Hello world")

    function()

This test will

@wrapt.decorator
def pass_through(wrapped, instance, args, kwargs):
    return wrapped(*args, **kwargs)

def test_wrapt(patch_wrapt_for_pycharm):
    @pass_through
    def function():
        print("Hello world")

    function()

[zyoung-rc]

@GrahamDumpleton I've narrowed it down further. It appears PyCharm does not like the ValueError raised. Changing it to an AttributeError seems to appease the debugger. So something like

from wrapt.wrappers import ObjectProxy

class ObjectProxyWithAttributeError(ObjectProxy):
    def __getattr__(self, name):
        if name == '__wrapped__':
            raise AttributeError('wrapper has not been initialised')

        return getattr(self.__wrapped__, name)

passes = ObjectProxyWithAttributeError("test")  # This works
fails = ObjectProxy("test")  # This raises an error from the PyCharm debugger

Something like this also works and would be more backwards compatible

class WrapperNotInitalisedError(AttributeError, ValueError):
    def __init__(self, msg:str = 'wrapper has not been initialised'):
        super().__init__(msg)

class ObjectProxyWithAttributeError(ObjectProxy):
    def __getattr__(self, name):
        if name == '__wrapped__':
            raise WrapperNotInitalisedError()

        return getattr(self.__wrapped__, name)

I hope this helps!

[hartym]

@zyoung-rc I confirm your fixture works for me, the small adjustments I made were to use scope="session" (should the patch be applied once per module ? I believe for now that it can be set for the whole testing session) and autouse=True (so I don't have to explicitely request it in each test). Also, I did put that code in a conftest.py file at root so pytest just get it. Thanks a lot for your work on that temporary fix !

[GrahamDumpleton]

Thanks for the confirmation that raising AttributeError avoids issue as suspected it might. The trick of using multiple inheritance so the exception type is both AttributeError and ValueError is also very interesting. I didn't even think about such a trick and resolves a concern I had of how to change the error type raised without potentially breaking existing code. I was thinking I would have to release a new version which still used ValueError, but via an environment variable flag switch it to AttributeError and allow people to use that to flesh out problems in real world applications before commit to switch to AttributeError as default.

As to monkey patching a temporary fix into existing code, I suspect that wrapt could be used to do that and have it monkey patch itself. I will need to play with that idea as a temporary fix.

[zyoung-rc]

@hartym I edited my comment to reflect the usage of scope="session". There was no particular reason for using module.

@GrahamDumpleton Glad to help! I would have written a PR, but, sadly, my C skills are non-existent.

[yangmeishu]

switch python to 3.10 can use