GrahamDumpleton
Improve ObjectProxy
There are many dunder methods documented here but are missing in ObjectProxy. Some of them like __call__ is defined in classes inheriting ObjectProxy. But for the sake of reusability of ObjectProxy, they should also be defined in it. Also, there are dunder methods like __next__ that are not even defined in inheriting classes. All of them should be defined in ObjectProxy.
It is not that simple.
Concerning __iter__, it is arguable that it was a mistake to add it to ObjectProxy in the first place as it has broken some code where people change behaviour based on a check of hasattr(obj, "__iter__"). That is, this check would always return True even though iter() would fail and raise an AttributeError. So in hindsight it perhaps shouldn't have been added, even if the hasattr check is sort of an anti Python and people should just capture errors when doing iteration itself.
As to __call__ is was never added because there callable() always existed for checking if something was callable and was clear that adding __call__ in base object proxy would cause issues straight away since people relied on callable(). That people were relying on hasattr(obj, "__iter__") was not so obvious.
Because of the issues with __iter__, this is why the asynchronous versions were never added later when that feature was added to Python.
As to __next__, what is the specific use case since __iter__ would call the __iter__ of the wrapped object and so the returned object which provides __next__ would not be a proxy object. Thus I don't see a direct reason why __next__ should appear in ObjectProxy as a default.
If you are overriding __iter__ on a custom object proxy to wrap the result when iter() is called, then you are already dealing with a mess of custom object proxies and so should be using such a custom object proxy to override __next__.
As to the statement that "there are many dunder methods documented but are missing in ObjectProxy", they would fall into two cases. The first is that they didn't exist in older Python versions when wrapt was first written. The second is that it isn't appropriate for them to be added on an object proxy, or at least not as default behaviour. In other words, would be more appropriate to add them in a custom object proxy.
So can you provide some example use case of what you are trying to do where you need __next__ to be proxied.
As to other dunder methods, list those you believe should be added by default on object proxy and provide specific examples so can understand why. In some cases where I learn of new dunder methods added in Python 3 they have been added, but only when asked and it makes sense to add them. I don't troll through each Python release looking for new dunder methods.
In order to solve problems of hasattr(obj, "__iter__") and callable(obj) you can define a function generating a ObjectProxy and before instantiating it, modify it depending on the needs of obj. Example for __call__:
def object_proxy(wrapped):
class ObjectProxy(with_metaclass(_ObjectProxyMetaType)): # type: ignore[misc]
__slots__ = "__wrapped__"
def __init__(self, wrapped):
object.__setattr__(self, "__wrapped__", wrapped)
# Python 3.2+ has the __qualname__ attribute, but it does not
# allow it to be overridden using a property and it must instead
# be an actual string object instead.
try:
object.__setattr__(self, "__qualname__", wrapped.__qualname__)
except AttributeError:
pass
# Python 3.10 onwards also does not allow itself to be overridden
# using a property and it must instead be set explicitly.
try:
object.__setattr__(self, "__annotations__", wrapped.__annotations__)
except AttributeError:
pass
def __self_setattr__(self, name, value):
object.__setattr__(self, name, value)
@property
def __name__(self):
return self.__wrapped__.__name__
@__name__.setter
def __name__(self, value):
self.__wrapped__.__name__ = value
@property
def __class__(self):
return self.__wrapped__.__class__
@__class__.setter
def __class__(self, value):
self.__wrapped__.__class__ = value
def __dir__(self):
return dir(self.__wrapped__)
def __str__(self):
return str(self.__wrapped__)
def __bytes__(self):
return bytes(self.__wrapped__)
def __repr__(self):
return f"<{type(self).__name__} at 0x{id(self):x} for {type(self.__wrapped__).__name__} at 0x{id(self.__wrapped__):x}>"
def __format__(self, format_spec):
return format(self.__wrapped__, format_spec)
def __reversed__(self):
return reversed(self.__wrapped__)
def __round__(self, ndigits=None):
return round(self.__wrapped__, ndigits)
def __mro_entries__(self, bases):
return (self.__wrapped__,)
def __lt__(self, other):
return self.__wrapped__ < other
def __le__(self, other):
return self.__wrapped__ <= other
def __eq__(self, other):
return self.__wrapped__ == other
def __ne__(self, other):
return self.__wrapped__ != other
def __gt__(self, other):
return self.__wrapped__ > other
def __ge__(self, other):
return self.__wrapped__ >= other
def __hash__(self):
return hash(self.__wrapped__)
def __nonzero__(self):
return bool(self.__wrapped__)
def __bool__(self):
return bool(self.__wrapped__)
def __setattr__(self, name, value):
if name.startswith("_self_"):
object.__setattr__(self, name, value)
elif name == "__wrapped__":
object.__setattr__(self, name, value)
try:
object.__delattr__(self, "__qualname__")
except AttributeError:
pass
try:
object.__setattr__(self, "__qualname__", value.__qualname__)
except AttributeError:
pass
try:
object.__delattr__(self, "__annotations__")
except AttributeError:
pass
try:
object.__setattr__(self, "__annotations__", value.__annotations__)
except AttributeError:
pass
elif name == "__qualname__":
setattr(self.__wrapped__, name, value)
object.__setattr__(self, name, value)
elif name == "__annotations__":
setattr(self.__wrapped__, name, value)
object.__setattr__(self, name, value)
elif hasattr(type(self), name):
object.__setattr__(self, name, value)
else:
setattr(self.__wrapped__, name, value)
def __getattr__(self, name):
# If we are being to lookup '__wrapped__' then the
# '__init__()' method cannot have been called.
if name == "__wrapped__":
raise WrapperNotInitializedError("wrapper has not been initialised")
return getattr(self.__wrapped__, name)
def __delattr__(self, name):
if name.startswith("_self_"):
object.__delattr__(self, name)
elif name == "__wrapped__":
raise TypeError("__wrapped__ must be an object")
elif name == "__qualname__":
object.__delattr__(self, name)
delattr(self.__wrapped__, name)
elif hasattr(type(self), name):
object.__delattr__(self, name)
else:
delattr(self.__wrapped__, name)
def __add__(self, other):
return self.__wrapped__ + other
def __sub__(self, other):
return self.__wrapped__ - other
def __mul__(self, other):
return self.__wrapped__ * other
def __truediv__(self, other):
return operator.truediv(self.__wrapped__, other)
def __floordiv__(self, other):
return self.__wrapped__ // other
def __mod__(self, other):
return self.__wrapped__ % other
def __divmod__(self, other):
return divmod(self.__wrapped__, other)
def __pow__(self, other, *args):
return pow(self.__wrapped__, other, *args)
def __lshift__(self, other):
return self.__wrapped__ << other
def __rshift__(self, other):
return self.__wrapped__ >> other
def __and__(self, other):
return self.__wrapped__ & other
def __xor__(self, other):
return self.__wrapped__ ^ other
def __or__(self, other):
return self.__wrapped__ | other
def __radd__(self, other):
return other + self.__wrapped__
def __rsub__(self, other):
return other - self.__wrapped__
def __rmul__(self, other):
return other * self.__wrapped__
def __rtruediv__(self, other):
return operator.truediv(other, self.__wrapped__)
def __rfloordiv__(self, other):
return other // self.__wrapped__
def __rmod__(self, other):
return other % self.__wrapped__
def __rdivmod__(self, other):
return divmod(other, self.__wrapped__)
def __rpow__(self, other, *args):
return pow(other, self.__wrapped__, *args)
def __rlshift__(self, other):
return other << self.__wrapped__
def __rrshift__(self, other):
return other >> self.__wrapped__
def __rand__(self, other):
return other & self.__wrapped__
def __rxor__(self, other):
return other ^ self.__wrapped__
def __ror__(self, other):
return other | self.__wrapped__
def __iadd__(self, other):
self.__wrapped__ += other
return self
def __isub__(self, other):
self.__wrapped__ -= other
return self
def __imul__(self, other):
self.__wrapped__ *= other
return self
def __itruediv__(self, other):
self.__wrapped__ = operator.itruediv(self.__wrapped__, other)
return self
def __ifloordiv__(self, other):
self.__wrapped__ //= other
return self
def __imod__(self, other):
self.__wrapped__ %= other
return self
def __ipow__(self, other): # type: ignore[misc]
self.__wrapped__ **= other
return self
def __ilshift__(self, other):
self.__wrapped__ <<= other
return self
def __irshift__(self, other):
self.__wrapped__ >>= other
return self
def __iand__(self, other):
self.__wrapped__ &= other
return self
def __ixor__(self, other):
self.__wrapped__ ^= other
return self
def __ior__(self, other):
self.__wrapped__ |= other
return self
def __neg__(self):
return -self.__wrapped__
def __pos__(self):
return +self.__wrapped__
def __abs__(self):
return abs(self.__wrapped__)
def __invert__(self):
return ~self.__wrapped__
def __int__(self):
return int(self.__wrapped__)
def __float__(self):
return float(self.__wrapped__)
def __complex__(self):
return complex(self.__wrapped__)
def __oct__(self):
return oct(self.__wrapped__)
def __hex__(self):
return hex(self.__wrapped__)
def __index__(self):
return operator.index(self.__wrapped__)
def __len__(self):
return len(self.__wrapped__)
def __contains__(self, value):
return value in self.__wrapped__
def __getitem__(self, key):
return self.__wrapped__[key]
def __setitem__(self, key, value):
self.__wrapped__[key] = value
def __delitem__(self, key):
del self.__wrapped__[key]
def __getslice__(self, i, j):
return self.__wrapped__[i:j]
def __setslice__(self, i, j, value):
self.__wrapped__[i:j] = value
def __delslice__(self, i, j):
del self.__wrapped__[i:j]
def __enter__(self):
return self.__wrapped__.__enter__()
def __exit__(self, *args, **kwargs):
return self.__wrapped__.__exit__(*args, **kwargs)
def __iter__(self):
return iter(self.__wrapped__)
def __copy__(self):
raise NotImplementedError("object proxy must define __copy__()")
def __deepcopy__(self, memo):
raise NotImplementedError("object proxy must define __deepcopy__()")
def __reduce__(self):
raise NotImplementedError("object proxy must define __reduce__()")
def __reduce_ex__(self, protocol):
raise NotImplementedError("object proxy must define __reduce_ex__()")
if callable(wrapped):
def __call__(self, *args, **kwargs):
return self.__wrapped__(*args, **kwargs)
ObjectProxy.__call__ = __call__
return ObjectProxy(wrapped)
Usage example:
So for example, if
inspect.isawaitable(obj) is the reason you didn't implement __await__, you can solve it the same way.
As to __next__, what is the specific use case
Proxying an iterator instead of a iterable:
As to other dunder methods, list those you believe should be added by default on object proxy
E.g There are __enter__ and __exit__ in ObjectProxy but their async versions are missing (__aenter__ and __aexit__). And if you decide to implement __next__, there is also __anext__. This is not the complete list. There are some other missing methods too.
You can do that inside of the __init__ method itself, but that technique is only of use for the pure Python implementation of ObjectProxy.
The pure Python implementation would rarely be used as it exists only as a fallback for when the C implementation isn't used for some reason.
In a C implementation of a Python object that sort of trick will not work because Python when it sees an object implemented in C, will only look in the tp_ slots of the C structure defining the type, and will ignore __iter__ and __call__ which are added as an attribute on the C object __dict__.
To be able to use the trick that works for the pure Python object implementation it would be necessary to change wrapt so that the existing C implementation of ObjectProxy is renamed to _ObjectProxy and then create a small ObjectProxy Python implementation shim that inherits from C based _ObjectProxy and in the Python implementation of the derived class __init__ then you could do it.
I will need to research though what the implications of doing this will be though on other dunder attributes which have weird rules such as __qualname__ and __annotations__, plus the meta class hacks. These may have to be replicated in that derived ObjectProxy for the C implementation as well as that in the C base class may no longer have the desired effect.
Although I expect it may be minimal enough that not a concern, adding the extra Python class deriving from C implementation may have extra overhead as a result.
Anyway, your response didn't answer the main question I put to you, which was the example you had where having __next__ as a default on ObjectProxy would be useful.
You can't do that inside __init__ because magic methods aren't automatically called when they are object attributes instead of class attributes. See this code:
class A:
def __init__(self):
def __call__(self, *args, **kwargs):
print("hello")
self.__call__ = __call__
x = A()
x()
This gives: TypeError: 'A' object is not callable
You also can't monkeypatch ObjectProxy inside __init__ because that change will be reflected on all ObjectProxy instances and it is possible that one instance should be callable and another shouldn't be. So that every ObjectProxy instance should have its own type.
your response didn't answer the main question I put to you
I don't have a real world example but i tried to answer you by giving an example of proxying an iterator. Isn't being unable to correctly proxy an iterator bad enough?
Lastly, you also didn't answer a question of mine. Why didn't you implement __aenter__ and __aexit__?
Using:
self.__call__ = __call__
would not have desired effect since that would set __call__ on the wrapped object and not the wrapper because of how __setattr__ is defined. Would have had to use object.__setattr__() instead.
Anyway, I keep forgetting that __call__ is one of those methods that needs to be set on the class and not on the instance even in pure Python code. Not all dunder methods necessarily need to be unless things have changed from early Python 2 days. The rules around them can be non obvious.
Rather than create a full class definition nested inside of a normal function, which would prohibit easily creating derived custom ObjectProxy classes, a better approach may be a minimal ObjectProxy derived class shim inside of a __new__ method that inherits from the cls argument passed to __new__. This will though need to be tolerant of a derived class already overriding __call__ and also want to ensure same type name comes through, the latter meaning can't use normal nested class and so construct it manually.
class A:
def __init__(self, wrapped):
self.__wrapped__ = wrapped
def __new__(cls, wrapped):
print("orig", cls)
namespace = {}
if callable(wrapped) and "__call__" not in dir(cls):
def __call__(self, *args, **kwargs):
print("base")
return self.__wrapped__(*args, **kwargs)
namespace["__call__"] = __call__
NewClass = type(cls.__name__, (cls,), namespace)
print("new", NewClass)
return super().__new__(NewClass)
class B(A):
def __call__(self, *args, **kwargs):
print("derived")
return self.__wrapped__(*args, *kwargs)
def func():
print("func")
a = A(func)
print(type(a))
a()
b = B(func)
print(type(b))
b()
Next problem to consider then is that __wrapped__ can technically be assigned to and the wrapped object changed after the wrapper has been initialised. One would have to document a caveat that the behaviour of the class for the wrapper would adopt what was required only for the wrapped object initially supplied when the wrapper is created, and overriding __wrapped__ would not change it if requirement was different. Since expect is unlikely anyone would be reassigning __wrapped__ this would be acceptable.
Oh, and regard question "why didn't you implement __aenter__ and __aexit__", because they never existed when wrapt was first written.
We can handle the case of assigning to __wrapped__ in __setattr__ so that class structure will adapt to the new object:
class A:
def __init__(self, wrapped):
self.__wrapped__ = wrapped
def __new__(cls, wrapped):
NewClass = type(cls)(cls.__name__, cls.__bases__, dict(cls.__dict__))
obj = super().__new__(NewClass)
if "__call__" in NewClass.__dict__:
obj._self_call_backup = type(obj).__call__
else:
def __call__(self, *args, **kwargs):
print("base")
return self.__wrapped__(*args, **kwargs)
obj._self_call_backup = __call__
obj.__init__(wrapped) # Must manually call since NewClass != cls
return obj
def __setattr__(self, name, value):
if name.startswith("_self_"):
object.__setattr__(self, name, value)
elif name == "__wrapped__":
if not ("__call__" in type(self).__dict__) and callable(value):
type(self).__call__ = self._self_call_backup
elif ("__call__" in type(self).__dict__) and not callable(value):
del type(self).__call__
object.__setattr__(self, name, value)
class B(A):
def __call__(self, *args, **kwargs):
print("derived")
return self.__wrapped__(*args, *kwargs)
def func():
pass
class C:
pass
def test():
a = A(func)
print(callable(a))
a.__wrapped__ = C()
print(callable(a))
a.__wrapped__ = func
print(callable(a))
a()
print()
a = B(func)
print(callable(a))
a.__wrapped__ = C()
print(callable(a))
a.__wrapped__ = func
print(callable(a))
a()
test()
Since expect is unlikely anyone would be reassigning
__wrapped__
Actually I'm working on a code that requires overloading operators like += and in there this feature is pretty important
@GrahamDumpleton I just found this issue from 2016 and it's still not fixed. If you are busy with other things these days, I can also update the python part of ObjectProxy and create a pull request. Would you want it or you already working on these?
I have already implemented a variant of dynamically adding a new class in __new__ as discussed to add __call__, __iter__, __next__, __aiter__ and __anext__. I also already added __aenter__, __aexit__ and __await__ direct to ObjectProxy.
I am still not keen on trying to change the class definition after the fact in rare case that __wrapped__ is assigned as there is more complexity to it than the code you suggested. Specifically, you should not delete any of the attributes if they were defined in a derived class, only if they were added dynamically. So have not made that change as yet and still thinking about it.
I haven't yet pushed changes up as I need to add more tests and have been busy with life stuff so haven't had time. I also still need to look more at dunder methods for matrix multiplication, which is another that currently isn't supported.
I modified my previous code snippet to check whether we are working with a dynamically added __call__ attribute or not before deleting it. This will not delete the __call__ attribute if it is defined by a derived class:
class A:
def __init__(self, wrapped):
self.__wrapped__ = wrapped
def __new__(cls, wrapped):
NewClass = type(cls)(cls.__name__, cls.__bases__, dict(cls.__dict__))
obj = super().__new__(NewClass)
if "__call__" in NewClass.__dict__:
obj._self_call_backup = type(obj).__call__
obj._self_delete_call = False
else:
def __call__(self, *args, **kwargs):
print("base")
return self.__wrapped__(*args, **kwargs)
obj._self_call_backup = __call__
obj._self_delete_call = True
obj.__init__(wrapped) # Must manually call since NewClass != cls
return obj
def __setattr__(self, name, value):
if name.startswith("_self_"):
object.__setattr__(self, name, value)
elif name == "__wrapped__":
if not ("__call__" in type(self).__dict__) and callable(value):
type(self).__call__ = self._self_call_backup
elif ("__call__" in type(self).__dict__) and not callable(value):
if self._self_delete_call:
del type(self).__call__
object.__setattr__(self, name, value)
class B(A):
def __call__(self, *args, **kwargs):
print("derived")
return self.__wrapped__(*args, *kwargs)
def func():
pass
class C:
pass
def test():
a = A(func)
print(callable(a))
a.__wrapped__ = C()
print(callable(a))
a.__wrapped__ = func
print(callable(a))
a()
print()
a = B(func)
print(callable(a))
a.__wrapped__ = C()
print(callable(a))
a.__wrapped__ = func
print(callable(a))
a()
test()
Is there any other problem with it now?
Great news you already implemented many dunder methods. But just like you gave the example of __matmul__, there are still many other unimplemented dunder methods. Example: __get__, __set__, __delete__
In respect of __get__ you would normally have:
obj.attr→ Python callsattr.__get__(obj, type(obj))
If you have basic proxying of __get__ then result is:
proxy.attr→ Python callsproxy.__get__(obj, type(obj))→ callswrapped.__get__(obj, type(obj))
The issue is that self in the wrapper functions refers to the proxy object, not the original wrapped object. This can break descriptor behaviour because:
- Identity issues: The descriptor receives the proxy as the
instanceparameter instead of the actual object it was designed to work with. - Type confusion:
ownerparameter might not match what the descriptor expects. - Binding context: Descriptors often rely on being bound to specific class hierarchies.
So I can't see that there is a generic proxy wrapper implementation one can have for special descriptor methods and binding protocol.
Binding for methods is complicated enough that wrapt has a special FunctionWrapper derived proxy type already to handle it all and that code is a mess of special cases.
In your sentence:
proxy.attr→ Python callsproxy.__get__(obj, type(obj))→ callswrapped.__get__(obj, type(obj))
I think you made a typo by writing proxy.attr instead of obj.attr, or even better, obj.proxy right?
Anyway, why would you pass self as the instance parameter? If you pass instance as the instance parameter and owner as the owner parameter there will be no problems. Like in this code:
class Descriptor:
def __get__(self, instance, owner = None):
print(instance)
class Proxy:
def __new__(cls, wrapped):
NewClass = type(cls)(cls.__name__, cls.__bases__, dict(cls.__dict__))
obj = super().__new__(NewClass)
if "__get__" in NewClass.__dict__:
obj._self_get_backup = type(obj).__get__
obj._self_delete_get = False
else:
def __get__(self, instance, owner = None):
return self.__wrapped__.__get__(instance, owner)
obj._self_get_backup = __get__
obj._self_delete_get = True
obj.__init__(wrapped) # Must manually call since NewClass != cls
return obj
def __init__(self, wrapped):
self.__wrapped__ = wrapped
def __setattr__(self, name, value):
if name.startswith("_self_"):
object.__setattr__(self, name, value)
elif name == "__wrapped__":
if not ("__get__" in type(self).__dict__) and hasattr(value, "__get__"):
type(self).__get__ = self._self_get_backup
elif ("__get__" in type(self).__dict__) and not hasattr(value, "__get__"):
if self._self_delete_get:
del type(self).__get__
object.__setattr__(self, name, value)
class A:
var1 = Descriptor()
var2 = Proxy(Descriptor())
@Proxy
def func1(self):
print(self)
a = A()
a.var1
a.var2
a.func1()
It prints the A object 3 times. No proxy object is printed.
That will teach me to bash out a response when running out the door and still groggy from a late night and lack of sleep. 🤣
So yes, I messed up badly and forgot the instance when doing binding comes from class the descriptor is used in. I thought I had something wrong, but was in a rush. Still not sure of a valid use case for having an object proxy wrapping a data descriptor. Non data descriptions, ie., binding behaviour as implemented by FunctionWrapper and as used in decorators, is more obvious.
Having an object proxy wrapping a data descriptor would be as easy as having an object proxy wrapping a non data descriptor. Just add __set__ and __delete__ dynamically in __new__ just like i showed how to for __get__. This way, you don't have to worry about what to do in __set__ and __delete__ in case __wrapped__ doesn't define them.
The nature of the changes discussed had already all been made, albeit differently to simplify things and ensure it could all be cleanly integrated with the C extension implementation. My comment was not about not knowing how to do it, but questioning what would be a real world use case for wrapping a proxy around a data descriptor. If you are still looking for something to do, try installing wrapt from the develop branch of the GitHub repo and test it.
Great! Now ObjectProxy supports most important dunder methods. What about these methods: __init_subclass__, __set_name__, __class_getitem__, __length_hint__, __trunc__, __floor__, __ceil__, __buffer__, __release_buffer__
And btw, when will you upload your last changes to pypi?
For the __init_subclass__ dunder method is a class method (not instance method) involved in class inheritance. It comes into play if you are inheriting from a class type and in the case of an object proxy has nothing to do with a wrapped object.
For the __set_name__ dunder method, although it could be argued that if include __get__ that it should also be included for completeness, there is a questionable need for it. The main purpose of ObjectProxy is when doing monkey patching, but if monkey patching a data descriptor, it would occur after the class definition has been processed and __set_name__ would already have been invoked by Python on the original data descriptor instance when the class definition was processed. Using ObjectProxy somehow inline to the class definition around a data descriptor doesn't make sense. If you really had to do that, you would use a custom class specific to the purpose where you would setup explicitly the descriptor methods you need and which need modifying somehow over original behaviour. Hard to see why you would use generic ObjectProxy in that inline case. All that said, __set_name__ was already implemented for FunctionWrapper, but then there is even less chance that it would have been getting used for that use case and was probably no need to add it.
The __class_getitem__ notionally is valid since a decorator could be applied to a class type. That said, it is not necessary to add it explicitly to ObjectProxy since Python doesn't appear to rely on it being declared on the type and so things work fine by virtue of __getattr__ chaining the attribute lookup to the wrapped object.
For the __length_hint__ dunder method, since automatically adding __iter__, could be argued that should be included. Because of how operator.length_hint() is implemented, it is forced to be on the class type. The problem though is that it can't be added as a default method on ObjectProxy which simply calls __length_hint__() of wrapped object as due to how operator.length_hint() is implemented, you would get an AttributeError exception raised when __length_hint__() does not exist on the wrapped object, rather than default length value being returned. This means would have to be dynamically added on shim type in __new__ but adding more and more methods that way gives me concern. Already not sure that mapping existing ObjectProxy to AutoObjectProxy is a good idea and may have to undo that and if people want it to automatically adapt, they perhaps need to opt in by using AutoObjectProxy explicitly. The cost of having a new class type created for each proxy instance may be too great as default which replaces existing behaviour.
The __trunc__, __floor__ and __ceil__ dunder methods may be okay to add as default methods direct on ObjectProxy as you need to be opting in to call them anyway with expectation that works. It is unlikely people would proxy access to literal numbers and even building custom number classes would be rare. So not sure whether in practice would ever get used.
The __buffer__ and __release_buffer__ dunder methods were only added in Python 3.12. No one has asked for these to be supported and enough of a corner case as to whether would come up in practice.
Overall, as much as one might think that trying to add every dunder method under the sun is a good idea, one has to be practical about it and should not run in and add methods where realistically the need for it would never arise. So I don't see an urgent need to add any of the above, except that since already added __get__() that could add __set_name__() for completeness.
As to when a final release will be made to PyPi, that will likely be some ways off. A release candidate can be done, but cautious about a final release. If not happy that everything is resolved and update will not cause issues by end of October, then unlikely to do a release until January as I can't easily support and make releases during December and last thing want is a severe issue to come up and a new release cannot be made quickly.
Right now there is still a bit of work to do to update tests and documentation for all the changes which have been made recently.
Have had to pull back on ObjectProxy being replaced with AutoObjectProxy. The memory overhead per instance with a new class type being created every time was way too much. Thus AutoObjectProxy is accessible as a separate type and you need to opt into using it, and it should only be used where you know you would have minimal instances of it. If you need special methods for iterator, descriptors, callable etc, you are still much better creating a custom object proxy base class with just what you need as then it being a separate single class type, you don't get per instance memory overhead from the class type.
adding more and more methods that way gives me concern
Since now we have a seperate class called AutoObjectProxy you can support __length_hint__ in it and not in the original ObjectProxy
Given that AutoObjectProxy is now opt in, I had already done that and also added __set_name__ as well.
I will close this issue if you confirm you will never implement __trunc__, __floor__, __ceil__, __buffer__, __release_buffer__
1 last thing: I recommend changing __mro_entries__ with this one:
def __mro_entries__(self, bases):
if not isinstance(self.__wrapped__, type) and hasattr(self.__wrapped__, "__mro_entries__"):
return self.__wrapped__.__mro_entries__(bases)
else:
return (self.__wrapped__,)
even building custom number classes would be rare
It just came to my mind: there is decimal from standard library which provides a custom numeric class Decimal. If someone does:
import decimal
import math
import wrapt
a = decimal.Decimal(10.1)
a = wrapt.ObjectProxy(a)
math.trunc(a) # Error
it will give error: TypeError: type ObjectProxy doesn't define __trunc__ method
I stumbled upon a strange behavior with in-place operators:
import wrapt
class A:
a = wrapt.ObjectProxy(10)
b = 10
x = A()
x.a += 1
x.b += 1
del x.a
del x.b
print(x.a)
print(x.b)
Those two don't print the same. They print 11 and 10 respectively. In order to solve this problem, i recommend changing in-place operators with something like this:
def __iadd__(self, other):
if hasattr(self.__wrapped__, "__iadd__"):
self.__wrapped__ += other
return self
else:
return type(self)(self.__wrapped__ + other) # This line probably contains bugs since current __new__ implementation inherits the class instead of copying and modifying its __dict__
This kind of behavior is also recommended in official python docs (falling back to __add__ when __iadd__ doesnt exists):
If __iadd__() does not exist, or if x.__iadd__(y) returns NotImplemented, x.__add__(y) and y.__radd__(x) are considered, as with the evaluation of x + y
You can't technically do:
return type(self)(self.__wrapped__ + other)
since a custom derived ObjectProxy type might be used which has a custom __init__() which accepts multiple arguments, or a single argument but where it isn't the final object to be wrapped, where in both cases arguments are interpreted in some other way to determine the final object to be wrapped, which is then passed to ObjectProxy base class __init__().
Have no idea what they are doing and why using ObjectProxy, but see example in:
- https://github.com/matthewwardrop/formulaic/blob/2be342cdf8cef8759ca22667eaa6291fe3fee526/formulaic/materializers/types/factor_values.py#L100
The original problem in this case is a strange one where will need to think about. The problem lies with how if you have class attribute, when accessing via an instance of the class, it can result in the class attribute being copied to the instance. Because though when object proxy is used it isn't a literal, it copies the object reference and so it isn't a disconnected instance like when copies a literal. Thus always updating the single instance which at that point is shared between class and instance.
I am not sure that changing behaviour based on whether __iadd__ exists or not is sensible thing to do, as that ultimately isn't where the issue arises, but in the Python behaviour where it copies class attributes to instance attributes automatically.
>>> class A:
... i = 0
...
>>> a = A()
>>> dir(a)
['__class__', '__delattr__', '__dict__', '__dir__', '__doc__', '__eq__', '__firstlineno__', '__format__', '__ge__', '__getattribute__', '__getstate__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__module__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__static_attributes__', '__str__', '__subclasshook__', '__weakref__', 'i']
>>> vars(a)
{}
>>> a.i += 1
>>> vars(a)
{'i': 1}
You can't even use a trick like:
import wrapt
class LiteralObjectProxy(wrapt.ObjectProxy):
pass
class A:
a = LiteralObjectProxy(10)
b = 10
def __setattr__(self, name, value):
if type(value) is LiteralObjectProxy:
value = LiteralObjectProxy(value.__wrapped__)
return object.__setattr__(self, name, value)
x = A()
x.a += 1
x.b += 1
print(x.a)
print(x.b)
del x.a
del x.b
print(x.a)
print(x.b)
because what Python is doing under the covers for += in this case is:
- makes a copy of the attribute
afrom the class to a local variable - apply
+=to the local copy of the variable - assign the local copy of the variable to the instance (which triggers
__setattr__)
If it had instead done:
- copy the attribute
afrom the class to the instance (which triggers__setattr__) - apply
+=to the variable on the instance
then one could use the trick in __setattr__ above, provided had used the special object proxy type.
One would have to dig into how instruction interpreter in Python works in this special case of making copy of class attribute before applying in place operator. Relying on whether __iadd__ (or other in-place dunder methods) exist seems a bit fragile. There must be a more logical way that interpreter deals with this for itself. Whether one can do anything about it for a proxy wrapper have no idea.
So this issue really relates only to builtin scalar types.
is_builtin_scalar = type(obj) in (int, float, str, bool, complex, bytes)
A list in Python also has __iadd__, but in this situation it will end up shared between class and instance even without object proxy.
One question with this though is why anyone would deliberately wrap a scalar type (likely intended as a constant), which appears on the class.
In the really strange circumstance that someone wants to wrap a scalar which is class attribute and a constant, they could do:
import wrapt
class ConstantObjectProxy(wrapt.ObjectProxy):
def __get__(self, instance, owner):
return ConstantObjectProxy(self.__wrapped__)
class B:
a = ConstantObjectProxy(10)
b = 10
x = B()
x.a += 1
x.b += 1
print(x.a)
print(x.b)
del x.a
del x.b
print(x.a)
print(x.b)
I just can't think of any good uses cases for it, except perhaps to try and track how many times the constant is accessed.