django-imagekit icon indicating copy to clipboard operation
django-imagekit copied to clipboard

Published 20 hours ago • verified publisher icon matthewwithanm

Problem using cache backend celery

Open jesushd12 opened this issue 5 years ago • 2 comments

Hello I´m getting this error while trying to Deferring Image Generation async with celery:

Internal Server Error: /client/business/3
Traceback (most recent call last):
  File "/Users/uppersky03/Documents/ProyectosMomentum/desarrollo/venv/lib/python3.6/site-packages/kombu/serialization.py", line 50, in _reraise_errors
    yield
  File "/Users/uppersky03/Documents/ProyectosMomentum/desarrollo/venv/lib/python3.6/site-packages/kombu/serialization.py", line 221, in dumps
    payload = encoder(data)
  File "/Users/uppersky03/Documents/ProyectosMomentum/desarrollo/venv/lib/python3.6/site-packages/kombu/serialization.py", line 350, in pickle_dumps
    return dumper(obj, protocol=pickle_protocol)
TypeError: can't pickle _thread._local objects

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/uppersky03/Documents/ProyectosMomentum/desarrollo/venv/lib/python3.6/site-packages/django/core/handlers/exception.py", line 35, in inner
    response = get_response(request)
  File "/Users/uppersky03/Documents/ProyectosMomentum/desarrollo/venv/lib/python3.6/site-packages/django/core/handlers/base.py", line 128, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "/Users/uppersky03/Documents/ProyectosMomentum/desarrollo/venv/lib/python3.6/site-packages/django/core/handlers/base.py", line 126, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/Users/uppersky03/Documents/ProyectosMomentum/desarrollo/venv/lib/python3.6/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
    return view_func(*args, **kwargs)
  File "/Users/uppersky03/Documents/ProyectosMomentum/desarrollo/venv/lib/python3.6/site-packages/rest_framework/viewsets.py", line 95, in view
    return self.dispatch(request, *args, **kwargs)
  File "/Users/uppersky03/Documents/ProyectosMomentum/desarrollo/venv/lib/python3.6/site-packages/rest_framework/views.py", line 494, in dispatch
    response = self.handle_exception(exc)
  File "/Users/uppersky03/Documents/ProyectosMomentum/desarrollo/venv/lib/python3.6/site-packages/rest_framework/views.py", line 454, in handle_exception
    self.raise_uncaught_exception(exc)
  File "/Users/uppersky03/Documents/ProyectosMomentum/desarrollo/venv/lib/python3.6/site-packages/rest_framework/views.py", line 491, in dispatch
    response = handler(request, *args, **kwargs)
  File "/Users/uppersky03/Documents/ProyectosMomentum/desarrollo/development/elmenu-heroku/elmenu-backend/elmenu/business/views.py", line 730, in partial_update
    self.perform_create(serializer)
  File "/Users/uppersky03/Documents/ProyectosMomentum/desarrollo/development/elmenu-heroku/elmenu-backend/elmenu/business/views.py", line 735, in perform_create
    serializer.save()
  File "/Users/uppersky03/Documents/ProyectosMomentum/desarrollo/venv/lib/python3.6/site-packages/rest_framework/serializers.py", line 209, in save
    self.instance = self.update(self.instance, validated_data)
  File "/Users/uppersky03/Documents/ProyectosMomentum/desarrollo/development/elmenu-heroku/elmenu-backend/elmenu/business/serializers.py", line 381, in update
    BusinessPhoto.objects.create(business=instance, photo=self.context.get("request").FILES.get(key))
  File "/Users/uppersky03/Documents/ProyectosMomentum/desarrollo/venv/lib/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/Users/uppersky03/Documents/ProyectosMomentum/desarrollo/venv/lib/python3.6/site-packages/django/db/models/query.py", line 417, in create
    obj.save(force_insert=True, using=self.db)
  File "/Users/uppersky03/Documents/ProyectosMomentum/desarrollo/venv/lib/python3.6/site-packages/django/db/models/base.py", line 729, in save
    force_update=force_update, update_fields=update_fields)
  File "/Users/uppersky03/Documents/ProyectosMomentum/desarrollo/venv/lib/python3.6/site-packages/django/db/models/base.py", line 769, in save_base
    update_fields=update_fields, raw=raw, using=using,
  File "/Users/uppersky03/Documents/ProyectosMomentum/desarrollo/venv/lib/python3.6/site-packages/django/dispatch/dispatcher.py", line 178, in send
    for receiver in self._live_receivers(sender)
  File "/Users/uppersky03/Documents/ProyectosMomentum/desarrollo/venv/lib/python3.6/site-packages/django/dispatch/dispatcher.py", line 178, in <listcomp>
    for receiver in self._live_receivers(sender)
  File "/Users/uppersky03/Documents/ProyectosMomentum/desarrollo/venv/lib/python3.6/site-packages/imagekit/specs/sourcegroups.py", line 33, in receiver
    fn(self, sender=sender, **kwargs)
  File "/Users/uppersky03/Documents/ProyectosMomentum/desarrollo/venv/lib/python3.6/site-packages/imagekit/specs/sourcegroups.py", line 102, in post_save_receiver
    attname)
  File "/Users/uppersky03/Documents/ProyectosMomentum/desarrollo/venv/lib/python3.6/site-packages/imagekit/specs/sourcegroups.py", line 124, in dispatch_signal
    signal.send(sender=source_group, source=file)
  File "/Users/uppersky03/Documents/ProyectosMomentum/desarrollo/venv/lib/python3.6/site-packages/django/dispatch/dispatcher.py", line 178, in send
    for receiver in self._live_receivers(sender)
  File "/Users/uppersky03/Documents/ProyectosMomentum/desarrollo/venv/lib/python3.6/site-packages/django/dispatch/dispatcher.py", line 178, in <listcomp>
    for receiver in self._live_receivers(sender)
  File "/Users/uppersky03/Documents/ProyectosMomentum/desarrollo/venv/lib/python3.6/site-packages/imagekit/registry.py", line 116, in source_group_receiver
    call_strategy_method(file, callback_name)
  File "/Users/uppersky03/Documents/ProyectosMomentum/desarrollo/venv/lib/python3.6/site-packages/imagekit/utils.py", line 166, in call_strategy_method
    fn(file)
  File "/Users/uppersky03/Documents/ProyectosMomentum/desarrollo/development/elmenu-heroku/elmenu-backend/elmenu/base/custom_strategy_imagekit.py", line 3, in on_source_saved
    file.generate()
  File "/Users/uppersky03/Documents/ProyectosMomentum/desarrollo/venv/lib/python3.6/site-packages/imagekit/cachefiles/__init__.py", line 94, in generate
    self.cachefile_backend.generate(self, force)
  File "/Users/uppersky03/Documents/ProyectosMomentum/desarrollo/venv/lib/python3.6/site-packages/imagekit/cachefiles/backends.py", line 134, in generate
    self.schedule_generation(file, force=force)
  File "/Users/uppersky03/Documents/ProyectosMomentum/desarrollo/venv/lib/python3.6/site-packages/imagekit/cachefiles/backends.py", line 163, in schedule_generation
    _celery_task.delay(self, file, force=force)
  File "/Users/uppersky03/Documents/ProyectosMomentum/desarrollo/venv/lib/python3.6/site-packages/celery/app/task.py", line 413, in delay
    return self.apply_async(args, kwargs)
  File "/Users/uppersky03/Documents/ProyectosMomentum/desarrollo/venv/lib/python3.6/site-packages/celery/app/task.py", line 536, in apply_async
    **options
  File "/Users/uppersky03/Documents/ProyectosMomentum/desarrollo/venv/lib/python3.6/site-packages/celery/app/base.py", line 737, in send_task
    amqp.send_task_message(P, name, message, **options)
  File "/Users/uppersky03/Documents/ProyectosMomentum/desarrollo/venv/lib/python3.6/site-packages/celery/app/amqp.py", line 554, in send_task_message
    **properties
  File "/Users/uppersky03/Documents/ProyectosMomentum/desarrollo/venv/lib/python3.6/site-packages/kombu/messaging.py", line 169, in publish
    compression, headers)
  File "/Users/uppersky03/Documents/ProyectosMomentum/desarrollo/venv/lib/python3.6/site-packages/kombu/messaging.py", line 252, in _prepare
    body) = dumps(body, serializer=serializer)
  File "/Users/uppersky03/Documents/ProyectosMomentum/desarrollo/venv/lib/python3.6/site-packages/kombu/serialization.py", line 221, in dumps
    payload = encoder(data)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/contextlib.py", line 99, in __exit__
    self.gen.throw(type, value, traceback)
  File "/Users/uppersky03/Documents/ProyectosMomentum/desarrollo/venv/lib/python3.6/site-packages/kombu/serialization.py", line 54, in _reraise_errors
    reraise(wrapper, wrapper(exc), sys.exc_info()[2])
  File "/Users/uppersky03/Documents/ProyectosMomentum/desarrollo/venv/lib/python3.6/site-packages/vine/five.py", line 178, in reraise
    raise value.with_traceback(tb)
  File "/Users/uppersky03/Documents/ProyectosMomentum/desarrollo/venv/lib/python3.6/site-packages/kombu/serialization.py", line 50, in _reraise_errors
    yield
  File "/Users/uppersky03/Documents/ProyectosMomentum/desarrollo/venv/lib/python3.6/site-packages/kombu/serialization.py", line 221, in dumps
    payload = encoder(data)
  File "/Users/uppersky03/Documents/ProyectosMomentum/desarrollo/venv/lib/python3.6/site-packages/kombu/serialization.py", line 350, in pickle_dumps
    return dumper(obj, protocol=pickle_protocol)
kombu.exceptions.EncodeError: can't pickle _thread._local objects

I think I configurate Imagekit properly like it says in the docs

IMAGEKIT_DEFAULT_CACHEFILE_BACKEND = 'imagekit.cachefiles.backends.Celery'
IMAGEKIT_DEFAULT_CACHEFILE_STRATEGY = 'base.custom_strategy_imagekit.ImagekitOnSaveStrategy'

Where ImagekitOnSaveStrategy

class ImagekitOnSaveStrategy(object):
    def on_source_saved(self, file):
        file.generate()

And my model look like this

    logo = models.ImageField(blank=True, null=True, upload_to=RandomFileName('businesses'))
    logo_thumbnail = ImageSpecField(source='logo',
                                      processors=[ResizeToFill(200, 200)],
                                      format='JPEG',
                                      options={'quality': 60})

When I run celery the task is recognize but I dont receive anything on the worker when I upload an image. . imagekit.cachefiles.backends._generate_file

I use redis for Cache and Celery with redis as a broker

Any help will be appreciate it!

My env: python 3.6 django-imagekit==4.0.2 celery==4.1.0

jesushd12 avatar Apr 01 '19 16:04 jesushd12

From the error what I see is that in the file object that should be pickled in order to be send via Redis to Celery there is a thread local object which is not picklable. The interesting thing is how this object is there.

I see that the issue is old (sorry for my slow response). Did you managed to fix it and how (or it is still unresolved)?

vstoykov avatar Jun 08 '19 21:06 vstoykov

Hi Matthew I couldnt resolve the problem yet im using optmistic strategy and assuming the first time load of the photo (waiting for cache creation) Do you think the buf can be fixed? Grettings

El sáb., 8 de jun. de 2019 5:48 PM, Venelin Stoykov < [email protected]> escribió:

From the error what I see is that in the file object that should be pickled in order to be send via Redis to Celery there is a thread local object which is not picklable. The interesting thing is how this object is there.

I see that the issue is old (sorry for my slow response). Did you managed to fix it and how (or it is still unresolved)?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/matthewwithanm/django-imagekit/issues/489?email_source=notifications&email_token=AB5L5FWDSMXBA6UCV4QFTM3PZQSMBA5CNFSM4HCYXEEKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXH7EIY#issuecomment-500167203, or mute the thread https://github.com/notifications/unsubscribe-auth/AB5L5FXMBELHLHVGYSHHLBLPZQSMBANCNFSM4HCYXEEA .

jesushd12 avatar Jun 12 '19 12:06 jesushd12