django-imagekit

django-imagekit copied to clipboard

I just tried to use imagekit on a project which stores media on s3, but I got the following error:

Template ValueError - The file cannot be reopened.

boto==2.39.0
Django==1.8.11
django-imagekit==3.3
django-s3-storage==0.9.8

Model:

image_thumbnail_300x200 = ImageSpecField(source='image',
                                      processors=[ResizeToFill(300, 200)],
                                      format='JPEG',
                                      options={'quality':90})

Template: {{ obj.image_thumbnail_300x200.url }}

This is probably a duplicate of #350

Probably it's related, but I'm not using imagekit template tags and I'm not trying to access width/height.

Ok then the issue is different, or the other issue is related to yours not the vise versa.

I'm kinda busy right now and don't have enough free time to investigate the issue and make tests for it.

I will appreciate any Pull Request which will have a solution for this.

On Django/1.9.5 and django-imagekit/3.3

I had the same issue with multiple ProcessedImageField fields in the same Model (Thus reusing the same original uploaded file stream).

https://github.com/matthewwithanm/django-imagekit/commit/13c92db760bc65f2436b93d764ee5ca2d3946e08 fix wasn't working since self.source.open() caused a seek(0) on a closed=True django.core.files.uploadedfile.InMemoryUploadedFile; which is prohibited.

Hackish/Dirty workaround for anyone in my situation... The fix was to override ProcessedImageFieldFile's save() this way:

class HackedProcessedImageFieldFile(ProcessedImageFieldFile):
    """
    Rendering multiple image from the same source resulted in I/O error due to an attempt from imagekit to
      rewind (seek(0)) the stream while it was closed.

    Looks like it's impossible to reopen a closed InMemoryFile (BytesIO Stream).
    """

    def save(self, name, content, save=True):

        # Copy original file/stream's content to a clone.
        initial = io.BytesIO()
        shutil.copyfileobj(self.file, initial)
        initial.seek(0)

        # Run preprocessor
        super(HackedProcessedImageFieldFile, self).save(name, content, save)

        # Copy clone over real file for reuse in next call.
        self.file.file = initial


class HackedProcessedImageField(ProcessedImageField):
    attr_class = HackedProcessedImageFieldFile


class MyModel(...):
    thumbnail = HackedProcessedImageField(...)

Stream is copied before being closed by whoever closes it and then re-applied over the original file.

@fabiocaccamo sorry for taking so long. Can you test the code in #380. I know that it does not sounds similar but I think that the problem you are facing can be related to the usage of NamedTemporaryFile which I removed in order to work with Django 1.10.

@raphael-riel I think that your issue is different despite looking similar. Probably another issue need to be opened but first I want to check if the problem that reported the original author of this issue is still present.

@fabiocaccamo forget about #380 I think that the fix that I made for #350 will work for you. Can you check #384 ?

I'm having this same problem and have finally managed to come up with full repro instructions. It does seem to be specific to s3 (and possibly django-s3-storage). I couldn't get it to repro using local storage. I didn't try another s3 backend (like django-storages).

This still repros on the develop branch (which includes #384 and #380).

I've put together a full test project for this, over here. See the README.md for full instructions, which I'm also pasting below.

---

S3 prep

First, create a new s3 bucket (ours is named iktest-media in this example). We'll need your access keys and bucket name available in your environment later, so go ahead and just export them now:

$ export AWS_ACCESS_KEY_ID=...
$ export AWS_SECRET_ACCESS_KEY=...
$ export AWS_S3_BUCKET_NAME=iktest-media  # replace with your bucket name

Now clone the iktest repo:

$ git clone https://github.com/mgalgs/iktest.git
$ cd iktest

and upload the IMG_20160609_183718.jpg file from this repo to your new bucket:

$ aws s3 cp IMG_20160609_183718.jpg s3://iktest-media/media/test_images/

Your bucket should now look like this:

$ aws s3 ls --recursive s3://iktest-media
2016-08-11 20:03:26      49093 media/test_images/IMG_20160609_183718.jpg

Django prep

Install everything in a fresh virtualenv and bootstrap some data:

$ virtualenv env
$ . env/bin/activate
$ pip install Django==1.9 django-s3-storage Pillow git+git://github.com/matthewwithanm/django-imagekit.git@develop
$ python manage.py migrate
$ python manage.py testdata

Running the app

You're now ready to run the dev server:

$ python manage runserver

Load http://localhost:8000 in your web browser. You'll notice that the first time you load the page it will complain with:

Internal Server Error: /
Traceback (most recent call last):
  File "/home/mgalgs/sites/iktest2/env/lib/python2.7/site-packages/django/core/handlers/base.py", line 149, in get_response
    response = self.process_exception_by_middleware(e, request)
  File "/home/mgalgs/sites/iktest2/env/lib/python2.7/site-packages/django/core/handlers/base.py", line 147, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/home/mgalgs/sites/iktest2/iktest/testapp/views.py", line 10, in index_view
    'widgets': TestWidget.objects.all(),
  File "/home/mgalgs/sites/iktest2/env/lib/python2.7/site-packages/django/shortcuts.py", line 67, in render
    template_name, context, request=request, using=using)
  File "/home/mgalgs/sites/iktest2/env/lib/python2.7/site-packages/django/template/loader.py", line 97, in render_to_string
    return template.render(context, request)
  File "/home/mgalgs/sites/iktest2/env/lib/python2.7/site-packages/django/template/backends/django.py", line 95, in render
    return self.template.render(context)
  File "/home/mgalgs/sites/iktest2/env/lib/python2.7/site-packages/django/template/base.py", line 206, in render
    return self._render(context)
  File "/home/mgalgs/sites/iktest2/env/lib/python2.7/site-packages/django/template/base.py", line 197, in _render
    return self.nodelist.render(context)
  File "/home/mgalgs/sites/iktest2/env/lib/python2.7/site-packages/django/template/base.py", line 988, in render
    bit = node.render_annotated(context)
  File "/home/mgalgs/sites/iktest2/env/lib/python2.7/site-packages/django/template/base.py", line 955, in render_annotated
    return self.render(context)
  File "/home/mgalgs/sites/iktest2/env/lib/python2.7/site-packages/django/template/defaulttags.py", line 220, in render
    nodelist.append(node.render_annotated(context))
  File "/home/mgalgs/sites/iktest2/env/lib/python2.7/site-packages/django/template/base.py", line 955, in render_annotated
    return self.render(context)
  File "/home/mgalgs/sites/iktest2/env/lib/python2.7/site-packages/django/template/base.py", line 1039, in render
    output = self.filter_expression.resolve(context)
  File "/home/mgalgs/sites/iktest2/env/lib/python2.7/site-packages/django/template/base.py", line 705, in resolve
    obj = self.var.resolve(context)
  File "/home/mgalgs/sites/iktest2/env/lib/python2.7/site-packages/django/template/base.py", line 846, in resolve
    value = self._resolve_lookup(context)
  File "/home/mgalgs/sites/iktest2/env/lib/python2.7/site-packages/django/template/base.py", line 887, in _resolve_lookup
    current = getattr(current, bit)
  File "/home/mgalgs/sites/iktest2/env/lib/python2.7/site-packages/imagekit/cachefiles/__init__.py", line 84, in url
    return self._storage_attr('url')
  File "/home/mgalgs/sites/iktest2/env/lib/python2.7/site-packages/imagekit/cachefiles/__init__.py", line 74, in _storage_attr
    existence_required.send(sender=self, file=self)
  File "/home/mgalgs/sites/iktest2/env/lib/python2.7/site-packages/django/dispatch/dispatcher.py", line 192, in send
    response = receiver(signal=self, sender=sender, **named)
  File "/home/mgalgs/sites/iktest2/env/lib/python2.7/site-packages/imagekit/registry.py", line 53, in existence_required_receiver
    self._receive(file, 'on_existence_required')
  File "/home/mgalgs/sites/iktest2/env/lib/python2.7/site-packages/imagekit/registry.py", line 61, in _receive
    call_strategy_method(file, callback)
  File "/home/mgalgs/sites/iktest2/env/lib/python2.7/site-packages/imagekit/utils.py", line 148, in call_strategy_method
    fn(file)
  File "/home/mgalgs/sites/iktest2/env/lib/python2.7/site-packages/imagekit/cachefiles/strategies.py", line 15, in on_existence_required
    file.generate()
  File "/home/mgalgs/sites/iktest2/env/lib/python2.7/site-packages/imagekit/cachefiles/__init__.py", line 93, in generate
    self.cachefile_backend.generate(self, force)
  File "/home/mgalgs/sites/iktest2/env/lib/python2.7/site-packages/imagekit/cachefiles/backends.py", line 108, in generate
    self.generate_now(file, force=force)
  File "/home/mgalgs/sites/iktest2/env/lib/python2.7/site-packages/imagekit/cachefiles/backends.py", line 95, in generate_now
    file._generate()
  File "/home/mgalgs/sites/iktest2/env/lib/python2.7/site-packages/imagekit/cachefiles/__init__.py", line 97, in _generate
    content = generate(self.generator)
  File "/home/mgalgs/sites/iktest2/env/lib/python2.7/site-packages/imagekit/utils.py", line 134, in generate
    content = generator.generate()
  File "/home/mgalgs/sites/iktest2/env/lib/python2.7/site-packages/imagekit/specs/__init__.py", line 153, in generate
    self.source.open()
  File "/home/mgalgs/sites/iktest2/env/lib/python2.7/site-packages/django/db/models/fields/files.py", line 81, in open
    self.file.open(mode)
  File "/home/mgalgs/sites/iktest2/env/lib/python2.7/site-packages/django/core/files/base.py", line 141, in open
    raise ValueError("The file cannot be reopened.")
ValueError: The file cannot be reopened.

If you reload your page it will work(!). However, the thumbnails that got generated won't actually be valid. Compare the urls from your web browser to:

$ aws s3 ls --recursive s3://iktest-media

FWIW, the issue I was experiencing (which I had postulated is the same issue being reported here) turned out to be due to the fact that django-s3-storage didn't support re-opening files. @etianen has since added support for that and the issue has gone away for me.

@fabiocaccamo, I noticed that you're using djangoe-s3-storage as well. You might want to try the 0.9.11 release to see if the problem goes away for you.

Bear in mind that there are plenty of file storage backends that don't allow files to be reopened. It's better to track down where the file is being closed and stop that happening.

@etianen actually if the custom storage does not allow files to be reopened then this is the problem of the custom storage because by default all files opened with Django's default storage can be reopened.

This is because Django implement only filesystem storage and django.core.files.File asume that it is opened from the filesystem and it's open method will work even if the underlying file was closed.

In order this to work in the open() method or more precisely in the _open() method of the custom storage need to returned instance of some subclass of File which need to know how to reopen itself if it was already closed.

https://github.com/django/django/blob/stable/1.10.x/django/core/files/storage.py#L299-L300 https://github.com/django/django/blob/stable/1.10.x/django/core/files/base.py#L135-L141

Also one more thing why files need to be closed https://github.com/matthewwithanm/django-imagekit/pull/335

P.S. @etianen Now I see that you are the owner of django-s3-storage and the fix you already provided is exaclty what I said above.

@mgalgs @etianen sorry that I was unable to help with resolving the issue in django-s3-storage but I was/am very bussy and does not have enough time to work on django-imagekit.

Now when the problem is discovered I need to find some time to release new version of django-imagekit with all fixes that are in master.

@mgalgs thanks for investigation and starting discussion on django-s3-starge issue tracker @etianen thanks for the fix in django-s3-storage

@mgalgs @etianen sorry that I was unable to help with resolving the issue in django-s3-storage but I was/am very bussy and does not have enough time to work on django-imagekit.

No need to apologize. Thanks for all your hard work.