tp_dictoffset semantics changed without a deprecation period

[encukou]

trafficstars

The tp_dictoffset field changed to “write-only” in 3.11, with docs going from 3.10's:

The real dictionary offset in an instance can be computed from a negative tp_dictoffset as follows: [code]

to:

The tp_dictoffset should be regarded as write-only. To get the pointer to the dictionary call PyObject_GenericGetDict().

The docs change was done to fix a release blocker, when it was clear that reverting the whole Py_TPFLAGS_MANAGED_DICT feature was not practical.

So, another “after-the-fact PEP 387 exception request”. More generally, we should think about how to make these less common.

[vstinner]

Python 3.11 got released in October 2022, so now it's too late to change it, no?

[encukou]

Probably? But someone could still add better porting docs, and there could be some process to ensure this doesn't happen again. (It looks like similar changes to PyTypeObject.tp_dict and PyLongObject.ob_digits are going into 3.12, which makes me uncomfortable.)

[gpshead]

As a data point, is anyone aware of any compatibility fallout from these specific changes?

[encukou]

No.

For context: The original change to tp_dict/Py_TPFLAGS_MANAGED_DICT did break at least pybind11 and mypy: https://github.com/python/cpython/issues/92678. The release blocker was fixed right before rc1.

The fix reduced the backwards-incompatibility to a docs-only change, deleting a calculation that is now invalid, but which can be easily replaced by calling PyObject_GenericGetDict. (The docs used to recommend _PyObject_GetDictPtr instead, which is underscore-prefixed but works well.) I'm not aware of breakage from that change. (It seems that the now-invalid calculation is present in Cython, but that might be dead code, or saved by debuggers handling data corruption gracefully. cc @scoder)

---

Or did you mean ob_digit? From a quick search of issues that got to me, that broke Cython-generated code, cypari2, fpylll2, mercurial, gmpy2, zbar. From a quick look:

• the fixes generally don't look future-proof: porting docs could be handy
• the broken projects were not using underscore-prefixed API -- it's unclear whether PEP-387 applies