librabbitmq Librabbitmq 2.0.0 with Python 3 gives TypeError: can't pickle memoryview objects

Librabbitmq 2.0.0 with Python 3 gives TypeError: can't pickle memoryview objects

Open sam-ghosh opened this issue 6 years ago • 6 comments

I am using the latest master branch of the git repo https://github.com/celery/librabbitmq and installing librabbitmq==2.0.0 for Python 3.6 by following the instructions in the readme

Using the development version

You can clone the repository by doing the following:

$ git clone git://github.com/celery/librabbitmq.git

Then install it by doing the following:

$ cd librabbitmq
$ make install        # or make develop

This works fine (after installing certain binaries for c compliation in the OS), but when I then make a small a+b add task and call it with add.delay(2,2) it fails with the following error. I looked up and saw that Celery 4 uses json as serializer, so clearly it is not because if pickle serialization

Changing from librabbitmq to pyamqp broker works normally
Same exact situation in both MacOS and Ubuntu 16

[2018-04-30 23:40:02,956: CRITICAL/MainProcess] Unrecoverable error: SystemError('<built-in method _basic_recv of Connection object at 0x1099ec5e8> returned a result with an error set',)
Traceback (most recent call last):
  File "/Users/somghosh/.virtualenvs/ctdb/lib/python3.6/site-packages/kombu/messaging.py", line 624, in _receive_callback
    return on_m(message) if on_m else self.receive(decoded, message)
  File "/Users/somghosh/.virtualenvs/ctdb/lib/python3.6/site-packages/celery/worker/consumer/consumer.py", line 570, in on_task_received
    callbacks,
  File "/Users/somghosh/.virtualenvs/ctdb/lib/python3.6/site-packages/celery/worker/strategy.py", line 145, in task_message_handler
    handle(req)
  File "/Users/somghosh/.virtualenvs/ctdb/lib/python3.6/site-packages/celery/worker/worker.py", line 221, in _process_task_sem
    return self._quick_acquire(self._process_task, req)
  File "/Users/somghosh/.virtualenvs/ctdb/lib/python3.6/site-packages/kombu/async/semaphore.py", line 62, in acquire
    callback(*partial_args, **partial_kwargs)
  File "/Users/somghosh/.virtualenvs/ctdb/lib/python3.6/site-packages/celery/worker/worker.py", line 226, in _process_task
    req.execute_using_pool(self.pool)
  File "/Users/somghosh/.virtualenvs/ctdb/lib/python3.6/site-packages/celery/worker/request.py", line 531, in execute_using_pool
    correlation_id=task_id,
  File "/Users/somghosh/.virtualenvs/ctdb/lib/python3.6/site-packages/celery/concurrency/base.py", line 155, in apply_async
    **options)
  File "/Users/somghosh/.virtualenvs/ctdb/lib/python3.6/site-packages/billiard/pool.py", line 1486, in apply_async
    self._quick_put((TASK, (result._job, None, func, args, kwds)))
  File "/Users/somghosh/.virtualenvs/ctdb/lib/python3.6/site-packages/celery/concurrency/asynpool.py", line 813, in send_job
    body = dumps(tup, protocol=protocol)
TypeError: can't pickle memoryview objects

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/somghosh/.virtualenvs/ctdb/lib/python3.6/site-packages/celery/worker/worker.py", line 203, in start
    self.blueprint.start(self)
  File "/Users/somghosh/.virtualenvs/ctdb/lib/python3.6/site-packages/celery/bootsteps.py", line 119, in start
    step.start(parent)
  File "/Users/somghosh/.virtualenvs/ctdb/lib/python3.6/site-packages/celery/bootsteps.py", line 370, in start
    return self.obj.start()
  File "/Users/somghosh/.virtualenvs/ctdb/lib/python3.6/site-packages/celery/worker/consumer/consumer.py", line 320, in start
    blueprint.start(self)
  File "/Users/somghosh/.virtualenvs/ctdb/lib/python3.6/site-packages/celery/bootsteps.py", line 119, in start
    step.start(parent)
  File "/Users/somghosh/.virtualenvs/ctdb/lib/python3.6/site-packages/celery/worker/consumer/consumer.py", line 596, in start
    c.loop(*c.loop_args())
  File "/Users/somghosh/.virtualenvs/ctdb/lib/python3.6/site-packages/celery/worker/loops.py", line 88, in asynloop
    next(loop)
  File "/Users/somghosh/.virtualenvs/ctdb/lib/python3.6/site-packages/kombu/async/hub.py", line 354, in create_loop
    cb(*cbargs)
  File "/Users/somghosh/.virtualenvs/ctdb/lib/python3.6/site-packages/kombu/transport/base.py", line 236, in on_readable
    reader(loop)
  File "/Users/somghosh/.virtualenvs/ctdb/lib/python3.6/site-packages/kombu/transport/base.py", line 218, in _read
    drain_events(timeout=0)
  File "/Users/somghosh/.virtualenvs/ctdb/lib/python3.6/site-packages/librabbitmq-2.0.0-py3.6-macosx-10.6-intel.egg/librabbitmq/__init__.py", line 227, in drain_events
    self._basic_recv(timeout)
SystemError: <built-in method _basic_recv of Connection object at 0x1099ec5e8> returned a result with an error set

May 01 '18 00:05 sam-ghosh

Hey - can you please explain this step: "This works fine (after installing certain binaries for c compliation in the OS)"

What did you install and how did you do it? Thanks!

May 11 '18 02:05 vgoklani

ref:

python3.6 can install py-amqp, it's ok for me.

This is a fork of amqplib which was originally written by Barry Pederson. It is maintained by the Celery project, and used by kombu as a pure python alternative when librabbitmq is not available. This library should be API compatible with librabbitmq.