Python 3.11: '_Code' object has no attribute 'co_positions'

[Yun-Kim]

I have a celery task that raises an exception:

@self.app.task
def fn_exception():
    raise Exception("Task class is failing")

When the task is applied, instead of being caught and sending a task failure signal, I get an attribute error from a biliard object: AttributeError: '_Code' object has no attribute 'co_positions'. co_positions is new in Python 3.11 for codeobjects, so I'm wondering if this is a billiards compatibility problem with Python 3.11 or if it's something else entirely with celery.

Here is the traceback:

.riot/venv_py3110_celery~505_redis~35/lib/python3.11/site-packages/celery/app/task.py:773: in apply
    ret = tracer(task_id, args, kwargs, request)
.riot/venv_py3110_celery~505_redis~35/lib/python3.11/site-packages/celery/app/trace.py:422: in trace_task
    I, R, state, retval = on_error(task_request, exc, uuid)
.riot/venv_py3110_celery~505_redis~35/lib/python3.11/site-packages/celery/app/trace.py:358: in on_error
    R = I.handle_error_state(
.riot/venv_py3110_celery~505_redis~35/lib/python3.11/site-packages/celery/app/trace.py:165: in handle_error_state
    return {
.riot/venv_py3110_celery~505_redis~35/lib/python3.11/site-packages/celery/app/trace.py:224: in handle_failure
    self._log_error(task, req, einfo)
.riot/venv_py3110_celery~505_redis~35/lib/python3.11/site-packages/celery/app/trace.py:252: in _log_error
    logger.log(policy.severity, policy.format.strip(), context,
../.pyenv/versions/3.11-dev/lib/python3.11/logging/__init__.py:1559: in log
    self._log(level, msg, args, **kwargs)
../.pyenv/versions/3.11-dev/lib/python3.11/logging/__init__.py:1634: in _log
    self.handle(record)
../.pyenv/versions/3.11-dev/lib/python3.11/logging/__init__.py:1644: in handle
    self.callHandlers(record)
../.pyenv/versions/3.11-dev/lib/python3.11/logging/__init__.py:1706: in callHandlers
    hdlr.handle(record)
../.pyenv/versions/3.11-dev/lib/python3.11/logging/__init__.py:978: in handle
    self.emit(record)
.riot/venv_py3110_mock_pytest700_pytest-mock_coverage_pytest-cov_opentracing_hypothesis/lib/python3.11/site-packages/_pytest/logging.py:331: in emit
    super().emit(record)
../.pyenv/versions/3.11-dev/lib/python3.11/logging/__init__.py:1118: in emit
    self.handleError(record)
../.pyenv/versions/3.11-dev/lib/python3.11/logging/__init__.py:1110: in emit
    msg = self.format(record)
../.pyenv/versions/3.11-dev/lib/python3.11/logging/__init__.py:953: in format
    return fmt.format(record)
.riot/venv_py3110_mock_pytest700_pytest-mock_coverage_pytest-cov_opentracing_hypothesis/lib/python3.11/site-packages/_pytest/logging.py:92: in format
    return super().format(record)
../.pyenv/versions/3.11-dev/lib/python3.11/logging/__init__.py:695: in format
    record.exc_text = self.formatException(record.exc_info)
../.pyenv/versions/3.11-dev/lib/python3.11/logging/__init__.py:645: in formatException
    traceback.print_exception(ei[0], ei[1], tb, None, sio)
../.pyenv/versions/3.11-dev/lib/python3.11/traceback.py:124: in print_exception
    te = TracebackException(type(value), value, tb, limit=limit, compact=True)
../.pyenv/versions/3.11-dev/lib/python3.11/traceback.py:688: in __init__
    self.stack = StackSummary._extract_from_extended_frame_gen(
../.pyenv/versions/3.11-dev/lib/python3.11/traceback.py:416: in _extract_from_extended_frame_gen
    for f, (lineno, end_lineno, colno, end_colno) in frame_gen:
../.pyenv/versions/3.11-dev/lib/python3.11/traceback.py:353: in _walk_tb_with_full_positions
    positions = _get_code_position(tb.tb_frame.f_code, tb.tb_lasti)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

code = <billiard.einfo._Code object at 0x7f4db0e40a10>, instruction_index = 428

    def _get_code_position(code, instruction_index):
        if instruction_index < 0:
            return (None, None, None, None)
>       positions_gen = code.co_positions()
E       AttributeError: '_Code' object has no attribute 'co_positions'

Note: I am running billard==3.6.4.0, celery==5.2.7, redis==3.5 with Python 3.11.

[auvipy]

try to down grade your python or use celery 5.3.0b1 with billiard 4.0.2

[Yun-Kim]

Hi @auvipy, I tried using celery==5.3.0b1, billiard==4.0.2 but am getting the same error 😢

I also checked billiard.einfo.py with version billiard==4.0.2 but it looks like the _Code class doesn't have the attribute co_positions as required by Python's traceback module. Will this be added anytime soon? Thanks!

[auvipy]

you can contribute to fix that

[Yun-Kim]

@eltonkl thanks for getting the fix! @auvipy When can we expect a release with the fix included?

[auvipy]

ping me again if not released in a week

[Yun-Kim]

@auvipy do you have any updates?

[fredley]

@auvipy another week, another ping (do say if you'd rather not be reminded).

[BoPeng]

Encountered the same problem so pinging @auvipy. :-)

[freemindcore]

@auvipy another week, another ping (do say if you'd rather not be reminded). Thanks!

[thedrow]

@auvipy We can release a new version of billiard without releasing Kombu/Celery yet.

[auvipy]

@auvipy We can release a new version of billiard without releasing Kombu/Celery yet.

100%

[auvipy]

I will release ASAP

[auvipy]

released a new version

[masarliev]

@auvipy celery 5.2.7 requireds billiard<4.0. Can you release 3.6.5.0(may be) with this fix?

[auvipy]

this release is for celery 5.3 mainly. we still do not back port python 3.11 support to celery 5.2.x

[sparrowt]

released a new version

That's https://github.com/celery/billiard/releases/tag/v4.1.0 for anyone else who hit this and is wondering how to get the fix.

However...

this release is for celery 5.3 mainly. we still do not back port python 3.11 support to celery 5.2.x

https://pypi.org/project/celery/#history suggests that all celery 5.3.x versions are still pre-release, so from your comment above it seems that there is no support (or plan for it) for python 3.11 in any current official 'release' of celery? 😢

[auvipy]

that can be used with 5.3beta versions as of now

[sparrowt]

Yes I see that, however we do not plan to use a beta version of celery in production - it's a shame that the current 5.2.x series is being left behind in terms of python 3.11 support.

[Nusnus]

Yes I see that, however we do not plan to use a beta version of celery in production - it's a shame that the current 5.2.x series is being left behind in terms of python 3.11 support.

I understand the need for Python 3.11 support my friend @sparrowt, and it will come, but we first need to concentrate on releasing the official Celery 5.3 release (no beta, no pre-releases, official), so we can close the circle on that aspect and move on forward. As much as Python 3.11 support would be awesome, releasing a stable 5.3 version is much higher on the list of priorities.

I am sure @auvipy and myself would happy to support any contribution towards adding Python 3.11, but at the moment our hands are full and we cannot depreioritize other issues to push this instead.

[christianbundy]

How to live dangerously (i.e. Python 3.11 with Celery 5.2):

diff --git a/.venv/lib/python3.11/site-packages/billiard/einfo.py b/.venv/lib/python3.11/site-packages/billiard/einfo.py
index 236f400c7..84ff7529d 100644
--- a/.venv/lib/python3.11/site-packages/billiard/einfo.py
+++ b/.venv/lib/python3.11/site-packages/billiard/einfo.py
@@ -24,6 +24,13 @@ class _Code(object):
         self.co_nlocals = code.co_nlocals
         self.co_stacksize = code.co_stacksize
         self.co_varnames = ()
+        if sys.version_info >= (3, 11):
+            self._co_positions = list(code.co_positions())
+
+    if sys.version_info >= (3, 11):
+        @property
+        def co_positions(self):
+            return self._co_positions.__iter__


 class _Frame(object):

THE SOFTWARE IS PROVIDED “AS IS” AND THE AUTHOR DISCLAIMS ALL WARRANTIES WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.