kombu
kombu copied to clipboard
celery
YAML not support memoryview
Hello I try to implement a celery protocol client based on kombu. When the result content_type is yaml, kombu throws an error:
File "/usr/local/lib/python3.5/dist-packages/kombu/serialization.py", line 50, in _reraise_errors
yield
File "/usr/local/lib/python3.5/dist-packages/kombu/serialization.py", line 263, in loads
return decode(data)
File "/usr/lib/python3/dist-packages/yaml/__init__.py", line 94, in safe_load
return load(stream, SafeLoader)
File "/usr/lib/python3/dist-packages/yaml/__init__.py", line 70, in load
loader = Loader(stream)
File "/usr/lib/python3/dist-packages/yaml/loader.py", line 24, in __init__
Reader.__init__(self, stream)
File "/usr/lib/python3/dist-packages/yaml/reader.py", line 85, in __init__
self.determine_encoding()
File "/usr/lib/python3/dist-packages/yaml/reader.py", line 124, in determine_encoding
self.update_raw()
File "/usr/lib/python3/dist-packages/yaml/reader.py", line 178, in update_raw
data = self.stream.read(size)
AttributeError: 'memoryview' object has no attribute 'read'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/kombu/messaging.py", line 618, in _receive_callback
decoded = None if on_m else message.decode()
File "/usr/local/lib/python3.5/dist-packages/kombu/message.py", line 192, in decode
self._decoded_cache = self._decode()
File "/usr/local/lib/python3.5/dist-packages/kombu/message.py", line 197, in _decode
self.content_encoding, accept=self.accept)
File "/usr/local/lib/python3.5/dist-packages/kombu/serialization.py", line 263, in loads
return decode(data)
File "/usr/lib/python3.5/contextlib.py", line 77, in __exit__
self.gen.throw(type, value, traceback)
File "/usr/local/lib/python3.5/dist-packages/kombu/serialization.py", line 54, in _reraise_errors
reraise(wrapper, wrapper(exc), sys.exc_info()[2])
File "/usr/local/lib/python3.5/dist-packages/vine/five.py", line 178, in reraise
raise value.with_traceback(tb)
File "/usr/local/lib/python3.5/dist-packages/kombu/serialization.py", line 50, in _reraise_errors
yield
File "/usr/local/lib/python3.5/dist-packages/kombu/serialization.py", line 263, in loads
return decode(data)
File "/usr/lib/python3/dist-packages/yaml/__init__.py", line 94, in safe_load
return load(stream, SafeLoader)
File "/usr/lib/python3/dist-packages/yaml/__init__.py", line 70, in load
loader = Loader(stream)
File "/usr/lib/python3/dist-packages/yaml/loader.py", line 24, in __init__
Reader.__init__(self, stream)
File "/usr/lib/python3/dist-packages/yaml/reader.py", line 85, in __init__
self.determine_encoding()
File "/usr/lib/python3/dist-packages/yaml/reader.py", line 124, in determine_encoding
self.update_raw()
File "/usr/lib/python3/dist-packages/yaml/reader.py", line 178, in update_raw
data = self.stream.read(size)
kombu.exceptions.DecodeError: 'memoryview' object has no attribute 'read'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "kombu_.py", line 91, in <module>
recv(task_id)
File "kombu_.py", line 85, in recv
conn.drain_events(timeout=0.1)
File "/usr/local/lib/python3.5/dist-packages/librabbitmq/__init__.py", line 227, in drain_events
self._basic_recv(timeout)
SystemError: <built-in method _basic_recv of Connection object at 0x7f29e74f4a68> returned a result with an error set
Finally, I replaced the yaml decoder:
import kombu.serialization
def yaml_load(data, *args, **kwargs):
if isinstance(data, memoryview):
data = data.tobytes()
return yaml.safe_load(data, *args, **kwargs)
kombu.serialization.registry.register('yaml', yaml.safe_dump, yaml_load,
content_type='application/x-yaml',
content_encoding='utf-8')
Had the same problem, the solution worked for me.
Celery 3.x Kombu 3.x PyYaml 3.x
