Celery - Task handler raised error: MaybeEncodingError ?

[Vido]

Hi guys!

Does sendgrid-django works well with Celery?

celery==4.0.2
sendgrid==3.6.3
sendgrid-django==4.0.4

# -*- coding: utf-8

import smtplib
from email.mime.text import MIMEText

from django.core.mail import EmailMultiAlternatives
from django.template.loader import get_template
from rotina_contabil.settings import SEND_SENDGRID_MAIL
from rotina_contabil.settings import EMAIL_USER, PASS_USER

def send_email(subject, _from, _to, template, data):

    template = get_template(template)
    html_content = template.render(data)

    mail = EmailMultiAlternatives(
        subject=subject,
        body=str(data),
        from_email=_from,
        to=[_to],
        reply_to=['Rotina Contábil <[email protected]>']
    )

    mail.attach_alternative(html_content, "text/html")
    mail.send()

[2017-07-03 11:54:03,442: ERROR/PoolWorker-2] Task utilidades.tasks.async_send_email[b64bf98f-77ad-4c87-85c6-1cf7dcbc2eb4] raised unexpected: BadRequestsError('HTTPError()',)
Traceback (most recent call last):
  File "/home/junkemen/Documents/pinewoods/99contatores/env34/lib/python3.4/site-packages/celery/app/trace.py", line 367, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/home/junkemen/Documents/pinewoods/99contatores/env34/lib/python3.4/site-packages/celery/app/trace.py", line 622, in __protected_call__
    return self.run(*args, **kwargs)
  File "/home/junkemen/Documents/pinewoods/99contatores/rotina_contabil/utilidades/tasks.py", line 8, in async_send_email
    send_email(*args, **kwargs)
  File "/home/junkemen/Documents/pinewoods/99contatores/rotina_contabil/utilidades/email.py", line 43, in send_email
    mail.send()
  File "/home/junkemen/Documents/pinewoods/99contatores/env34/lib/python3.4/site-packages/django/core/mail/message.py", line 292, in send
    return self.get_connection(fail_silently).send_messages([self])
  File "/home/junkemen/Documents/pinewoods/99contatores/env34/lib/python3.4/site-packages/sgbackend/mail.py", line 68, in send_messages
    self.sg.client.mail.send.post(request_body=mail)
  File "/home/junkemen/Documents/pinewoods/99contatores/env34/lib/python3.4/site-packages/python_http_client/client.py", line 211, in http_request
    return Response(self._make_request(opener, request))
  File "/home/junkemen/Documents/pinewoods/99contatores/env34/lib/python3.4/site-packages/python_http_client/client.py", line 145, in _make_request
    raise exc
python_http_client.exceptions.BadRequestsError: HTTPError()
[2017-07-03 11:54:03,472: ERROR/MainProcess] Task handler raised error: <MaybeEncodingError: Error sending result: '"(1, <ExceptionInfo: BadRequestsError('HTTPError()',)>, None)"'. Reason: ''PicklingError("Can\'t pickle <class \'python_http_client.exceptions.BadRequestsError\'>: it\'s not the same object as python_http_client.exceptions.BadRequestsError",)''.>
Traceback (most recent call last):
  File "/home/junkemen/Documents/pinewoods/99contatores/env34/lib/python3.4/site-packages/billiard/pool.py", line 363, in workloop
    put((READY, (job, i, result, inqW_fd)))
  File "/home/junkemen/Documents/pinewoods/99contatores/env34/lib/python3.4/site-packages/billiard/queues.py", line 366, in put
    self.send_payload(ForkingPickler.dumps(obj))
  File "/home/junkemen/Documents/pinewoods/99contatores/env34/lib/python3.4/site-packages/billiard/reduction.py", line 56, in dumps
    cls(buf, protocol).dump(obj)
billiard.pool.MaybeEncodingError: Error sending result: '"(1, <ExceptionInfo: BadRequestsError('HTTPError()',)>, None)"'. Reason: ''PicklingError("Can\'t pickle <class \'python_http_client.exceptions.BadRequestsError\'>: it\'s not the same object as python_http_client.exceptions.BadRequestsError",)''.

[sklarsa]

I'm actually working on a similar package (https://github.com/sklarsa/django-sendgrid-v4) and am experiencing a billiard-related error in a celery task. I'm digging into it a little deeper now to see if I can find anything:

Traceback (most recent call last):
  File "/Users/steven/predata/venv/lib/python2.7/site-packages/billiard/pool.py", line 1175, in mark_as_worker_lost
    human_status(exitcode)),
WorkerLostError: Worker exited prematurely: signal 11 (SIGSEGV).

[sklarsa]

It actually looks like my error is matplotlib-related (different from yours)... Are all of your celery workers running the same version of python_http_client?

[Vido]

@sklarsa , I just ditched sendgrid-django in favor of sendgrid-python.

[jayhale]

@Vido there doesn't appear to be an issue with compatibility between sendgrid-django and celery. Instead, your API request failed with error 400:

python_http_client.exceptions.BadRequestsError: HTTPError()

...and then the error class failed to pickle for some reason:

billiard.pool.MaybeEncodingError: Error sending result: '"(1, <ExceptionInfo: BadRequestsError('HTTPError()',)>, None)"'. Reason: ''PicklingError("Can\'t pickle <class \'python_http_client.exceptions.BadRequestsError\'>: it\'s not the same object as python_http_client.exceptions.BadRequestsError",)''.

This isn't a sendgrid-django error. Recommend closing.