django-imagekit

django-imagekit copied to clipboard

Getting this issue when creating a new image. I simply have a few specs on an ImageField like this:

    big_image = ImageSpecField(
        [SmartResize(width=900, height=600)],
        source='original_image',
        format='JPEG',
        options={'quality': 75}
    )

Exception:

ValueError` at /api/v1/pictures/pictures/

I/O operation on closed file.
Request Method: POST
Request URL:    https://foobar.com/api/v1/pictures/pictures/
Django Version: 1.10.2
Exception Type: ValueError
Exception Value:    
I/O operation on closed file.
Exception Location: /home/foobar/.virtualenvs/foobar/local/lib/python2.7/site-packages/imagekit/cachefiles/__init__.py in _generate, line 102

Later part of the stack trace, somehow the "File" gets set to none?:

/home/foobar/.virtualenvs/foobar/local/lib/python2.7/site-packages/imagekit/cachefiles/backends.py in generate_now
            file._generate()
Local Vars

Variable    Value
file    
<ImageCacheFile: CACHE/images/pictures/2016/10/25/foobar.jpg>
force   
False
self    
<imagekit.cachefiles.backends.Simple object at 0x7f9e2e91ce50>
/home/foobar/.virtualenvs/foobar/local/lib/python2.7/site-packages/imagekit/cachefiles/__init__.py in _generate
        content.seek(0)
Local Vars

Variable    Value
actual_name 
u'CACHE/images/pictures/2016/10/25/foobar.jpg'
content 
<File: None>
self    
<ImageCacheFile: CACHE/images/pictures/2016/10/25/foobar.jpg>

I have a setup with S3 and this started happening when I updated the boto library to boto3. Reverting back seemed to help. I can provide more info if needed.

Linked to https://github.com/matthewwithanm/django-imagekit/issues/369 or https://github.com/matthewwithanm/django-imagekit/issues/350 or https://github.com/matthewwithanm/django-imagekit/issues/365 ?

Finally some clue. It looks like smething changed in boto3 and the way how django imagekit is playing with files is not very usefull for boto3.

Currently I'm very busy with a big project and does not have enough time to investigate the problem with boto.

If you can investigate it and prepare a pull request I will be be grateful.

I will definitely spend some time to investigate, but this feels like it goes beyond my capabilities. Nonetheless, will keep this ticket updated if I find anything.

Some findings:

• It sometimes breaks with 1 ImageSpecField and default settings
• It sometimes breaks when having 2 ImageSpecFields and default settings
• It sometimes breaks when having 1 ImageSpecField and IMAGEKIT_DEFAULT_CACHEFILE_STRATEGY = 'imagekit.cachefiles.strategies.Optimistic'

The following line closes the file:

actual_name = self.storage.save(self.name, content)

Narrowed it down to https://github.com/boto/boto3/blob/develop/boto3/s3/inject.py#L372 > https://github.com/boto/s3transfer/blob/develop/s3transfer/manager.py#L242 where the file gets closed (I assume).

Could it be that boto3 closes the file because it uses a future? :returns: Transfer future representing the upload

The quickfix would be to simply re-open the file again, but that might be bad design?

@philippeluickx After this fix in django-s3-storage https://github.com/etianen/django-s3-storage/commit/0286570e5afbfa6909fa65d95d0fb49ed1e45ebc this issue must be gone. Check with django-s3-storrage==0.9.11

@vstoykov I am however using (the more popular / mature) https://github.com/jschneier/django-storages

Now when I see in django-storages source code I see that the storage does not support reopening of the file. Also as stated in this issue https://github.com/jschneier/django-storages/issues/64 there may be a problem with closed attribute to not return True when the file is closed.

Is it possible to test with django-s3-storage to see if your problem is still there or not? If it is gone then the issue in django-storages need to be opened for this.

Here's my monkeypatch.py (imported by urls.py) that works around it the dirty way for now:

import logging


LOG = logging.getLogger(__name__)


def PATCH_IMAGEKIT_IMAGECACHEFILE_GENERATE():
    """
    Only needed until this issue gets fixed.
    <https://github.com/matthewwithanm/django-imagekit/issues/391>

    """
    from imagekit.cachefiles import ImageCacheFile
    from imagekit.utils import generate
    from django.core.files import File
    from imagekit.utils import get_logger

    def PATCHED_GENERATE(self):
        # Generate the file
        content = generate(self.generator)

        # PATCHED
        def PATCHED_CLOSE():
            """Protect file from being closed"""
            LOG.warning('patched close() called - ignoring', stack_info=False)

        ORIG_CLOSE = content.close
        content.close = PATCHED_CLOSE

        # Here content.close() gets called which is what we don't want
        actual_name = self.storage.save(self.name, content)

        # restore
        content.close = ORIG_CLOSE
        # /PATCHED

        # We're going to reuse the generated file, so we need to reset the pointer.

        content.seek(0)

        # Store the generated file. If we don't do this, the next time the
        # "file" attribute is accessed, it will result in a call to the storage
        # backend (in ``BaseIKFile._get_file``). Since we already have the
        # contents of the file, what would the point of that be?
        self.file = File(content)

        if actual_name != self.name:
            get_logger().warning(
                'The storage backend %s did not save the file with the'
                ' requested name ("%s") and instead used "%s". This may be'
                ' because a file already existed with the requested name. If'
                ' so, you may have meant to call generate() instead of'
                ' generate(force=True), or there may be a race condition in the'
                ' file backend %s. The saved file will not be used.' % (
                    self.storage,
                    self.name, actual_name,
                    self.cachefile_backend
                )
            )

    # Apply main patch
    LOG.warning('PATCHING ImageCacheFile._generate from imagekit')
    ImageCacheFile._generate = PATCHED_GENERATE


PATCH_IMAGEKIT_IMAGECACHEFILE_GENERATE()

Works with:

boto3==1.4.0
botocore==1.4.78
Django==1.10.2
django-imagekit==3.3
django-storages==1.5.1

@jkbrzt can you test with latest version of django-s3-storage instead of django-storages to see if your problem still persist without monkey patching? From what I see the problem is not in django-imagekit but in the underlying storage backend that do not support reopening (which by default is available in the default storage backend from Django). With your "patch" you can leak open files and at the end to have this problem:

IOError: [Errno 24] Too many open files

More info here https://github.com/matthewwithanm/django-imagekit/pull/335

@vstoykov unfortunately, this specific project is pretty tied to django-storages so django-s3-storage is not an option at the moment.

As for the Too many open files issue, I assumed django-imagekit closes the files whenever it's finished with them so thanks for bringing it up.

Same error happens on saving inlineformset with ImageSpecField. I have CreateView with form and inlineformset. Formset has 3 ImageSpecField fields and this error occurs on saving the second one.

@shapoglyk are you also using django-storages and storing files on S3 or your case is different? If it is the same can you replace django-storages with django-s3-storage?

@vstoykov , no I am using default django file storage.

That's interesting. Can you install from git to confirm that the issue is still there or is fixed but unreleased?

As philippeluickx pointed out, this seems to be boto3 related as the upload_fileobj closes the file you pass to it.

https://github.com/boto/boto3/issues/929 https://github.com/boto/s3transfer/issues/80

I managed to get around this by passing a clone of the file object to boto3 that it can close and cleanup but keeping the original around & unclosed.

class CustomS3Boto3Storage(S3Boto3Storage):
    """
    This is our custom version of S3Boto3Storage that fixes a bug in boto3 where the passed in file is closed upon upload.

    https://github.com/boto/boto3/issues/929
    https://github.com/matthewwithanm/django-imagekit/issues/391
    """
    
    def _save_content(self, obj, content, parameters):
        """
        We create a clone of the content file as when this is passed to boto3 it wrongly closes
        the file upon upload where as the storage backend expects it to still be open
        """
        # Seek our content back to the start
        content.seek(0, os.SEEK_SET)

        # Create a temporary file that will write to disk after a specified size
        content_autoclose = SpooledTemporaryFile()

        # Write our original content into our copy that will be closed by boto3
        content_autoclose.write(content.read())

        # Upload the object which will auto close the content_autoclose instance
        super(CustomS3Boto3Storage, self)._save_content(obj, content_autoclose, parameters)
        
        # Cleanup if this is fixed upstream our duplicate should always close        
        if not content_autoclose.closed:
            content_autoclose.close()

@leonsmith I confirm that your custom storage class works.

I'll be using it until a fix is made.

Thanks.

Hi guys. Can you check if the latest changes in develop (changes from #405) will fix the issue for you?

@charlesthk @shapoglyk @jkbrzt @philippeluickx

@vstoykov I have the same problem with:

django==1.10.5 django-storages==1.5.2 boto3==1.4.4 django-imagekit==4.0

@CedricCarrard thank you for reporting. Can you paste some traceback.

@vstoykov I have use imagekit.models.ProcessedImageField I have this problem when I submit a form with my image upload on s3

"/usr/local/lib/python3.4/site-packages/django/template/base.py"
current = current[bit]
TypeError: 'ImageCacheFile' object is not subscriptable
File "/usr/local/lib/python3.4/site-packages/imagekit/cachefiles/strategies.py", line 15, in on_existence_required file.generate()
File "/usr/local/lib/python3.4/site-packages/imagekit/cachefiles/__init__.py", line 93, in generate
self.cachefile_backend.generate(self, force)
File "/usr/local/lib/python3.4/site-packages/imagekit/cachefiles/backends.py", line 109, in generate
self.generate_now(file, force=force)
File "/usr/local/lib/python3.4/site-packages/imagekit/cachefiles/backends.py", line 96, in generate_now 
file._generate()
File "/usr/local/lib/python3.4/site-packages/imagekit/cachefiles/__init__.py", line 102, in _generate
content.seek(0)
File "/usr/local/lib/python3.4/tempfile.py", line 417, in func_wrapper
return func(*args, **kwargs)
ValueError: seek of closed file

After the submit of my form I use

{% generateimage 'apptest:Mythumbmail' source=object.picture %}

The problem comes from this

@vstoykov

I fixed my problem:

IMAGEKIT_DEFAULT_CACHEFILE_STRATEGY = 'app.example.imagegenerators.FixJustInTime'

class FixJustInTime:

    def on_content_required(self, file):
        try:
            file.generate()
        except:
            pass

    def on_existence_required(self, file):
        try:
            file.generate()
        except:
            pass

You have a better solution ?

@CedricCarrard did you try the workaround from https://github.com/matthewwithanm/django-imagekit/issues/391#issuecomment-275367006 ?

Your solution looks too hacky but if it works for you probably you can still use it.

I'm experiencing the same issue. It occurs when I add an ImageSpecField to a model that already has a ProcessedImageField.

self = <ImageCacheFile: CACHE/avatars/a0cbd7767bf54e649f4952e8c3407418/9dd51a8dd80342ae7576d1c376b51e43.jpg>

    def _generate(self):
        # Generate the file
        content = generate(self.generator)

        actual_name = self.storage.save(self.name, content)

        # We're going to reuse the generated file, so we need to reset the pointer.
>       content.seek(0)
E       ValueError: I/O operation on closed file.

actual_name = 'CACHE/avatars/a0cbd7767bf54e649f4952e8c3407418/9dd51a8dd80342ae7576d1c376b51e43.jpg'
content    = <File: None>
self       = <ImageCacheFile: CACHE/avatars/a0cbd7767bf54e649f4952e8c3407418/9dd51a8dd80342ae7576d1c376b51e43.jpg>

../../../.virtualenvs/myapp/lib/python3.5/site-packages/imagekit/cachefiles/__init__.py:102: ValueError

The workaround posted in https://github.com/matthewwithanm/django-imagekit/issues/391#issuecomment-275367006 works for me. Thanks!

Just another note - I'm having the same issue.

@mgrdcm Is the workaround in https://github.com/matthewwithanm/django-imagekit/issues/391#issuecomment-275367006 works for you?

Encountered this problem on latest django/imagekit/storages, can confirm workaround #391 worked for me.

https://github.com/matthewwithanm/django-imagekit/issues/391#issuecomment-275367006 worked for me too.

I was struggling all day long with this, and got it working thanks to #391.

I actually had to do a content_autoclose.seek(0) right after the write, because I was using client.upload_fileobj(), but it worked.

Thanks dude.

As I see in the comments of the issue in boto3 (https://github.com/boto/s3transfer/issues/80#issuecomment-296654643) someone have changed django-storages with django-s3-storage (https://github.com/etianen/django-s3-storage). You can also try it to see if it will work better for you

I thought Wagtail was causing this issue. Specially when uploading user images. Thanks to #391 I was able to move past it:

Django==1.8.18 wagtail==1.12 boto3==1.4.6 botocore==1.6.5 django-storages==1.6.5

Still the same problem with newest versions:

boto3==1.4.7 botocore==1.7.29 django-imagekit==4.0.1 django-storages==1.6.5 Django==1.11.6

Any fix or update?

@mayait what versions are using (imagekit, django). Did you use custom storage if yes which one (with version). If it is related to S3 probably you read that one https://github.com/boto/s3transfer/issues/80.

@vstoykov @leonsmith Thank you very much for the fix 👍

Running into this issue as well, even when just using DEFAULT_FILE_STORAGE = 'djet.files.InMemoryStorage' during unit testing (so nothing related to boto or s3 libs) if I have more than one ImageSpecField in my model.

And the bug only happens to me on django-imagekit==4.0.2 so I guess its related to this file leaking fix: https://github.com/matthewwithanm/django-imagekit/commit/6ee931398f3003c5e7a1a35b0ca8a99c49ac3012

My temporary fix is to use 4.0.1 instead and hope the file leaking issue doesn't bite me in the ass, haha. Looks like something still needs fixing in this library.

@adamJLev On a first glance as I see how djet.files.InMemoryStorage is implemented it does not support reopening of files by default. Your option may be to use ContentFile because it's close() method does not actually closes the underlying in memory file. If this fixes the problem for you with django-imagekit==4.0.2 then probably PR to djet need to be made to use ContentFiles instead of InMemoryUploadedFile. Or probably they can still use InMemoryUploadedFile but yo use ContentFile instead of io.BytesIO for the stream.

This is a complete blocker for us and the above mentioned fixes don't work. Issue is identical to original poster.

We're using django-imagekit==4.0.2 and django-storages[boto3]==1.7.1

Maybe this will help someone. Leon's suggest pointed the way, but here's what worked for me in a flask app where I implemented S3.

    s3 = boto3.client("s3")
    bucket_name = bucket_name or current_app.config["AWS_S3_BUCKET"]

    # Need to clone the file because boto3 autocloses the file after uploading to s3,
    # preventing further use of the uploaded file.
    file_s3 = SpooledTemporaryFile()
    file_s3.write(file.read())
    file_s3.seek(0)

    s3.upload_fileobj(
        file_s3,
        bucket_name,
        path,
        ExtraArgs={"ACL": "public-read", "ContentType": file.content_type},
    )

    if not file_s3.closed:
        file_s3.close()

Seeking back to zero on the copied file AFTER its written is what made it work for me.

I still have this problem - I'm getting the I/O error when trying to show a scaled image derived from my standard image_field.

Can confirm the workaround suggested by @leonsmith 391# worked for me, here the stack I used

Django==1.11.5 django-storages==1.6.5 botocore==1.12.91 boto3==1.9.143 django-imagekit==4.0

As philippeluickx pointed out, this seems to be boto3 related as the upload_fileobj closes the file you pass to it.

boto/boto3#929 boto/s3transfer#80

I managed to get around this by passing a clone of the file object to boto3 that it can close and cleanup but keeping the original around & unclosed.

class CustomS3Boto3Storage(S3Boto3Storage):
    """
    This is our custom version of S3Boto3Storage that fixes a bug in boto3 where the passed in file is closed upon upload.

    https://github.com/boto/boto3/issues/929
    https://github.com/matthewwithanm/django-imagekit/issues/391
    """
    
    def _save_content(self, obj, content, parameters):
        """
        We create a clone of the content file as when this is passed to boto3 it wrongly closes
        the file upon upload where as the storage backend expects it to still be open
        """
        # Seek our content back to the start
        content.seek(0, os.SEEK_SET)

        # Create a temporary file that will write to disk after a specified size
        content_autoclose = SpooledTemporaryFile()

        # Write our original content into our copy that will be closed by boto3
        content_autoclose.write(content.read())

        # Upload the object which will auto close the content_autoclose instance
        super(CustomS3Boto3Storage, self)._save_content(obj, content_autoclose, parameters)
        
        # Cleanup if this is fixed upstream our duplicate should always close        
        if not content_autoclose.closed:
            content_autoclose.close()

Where this code paste in django projects? Is it paste in storage_backend.py?

If anyone needs this is an updated version of @leonsmith custom class for use with the latest django-storages. They've removed _save_content class.

class CustomS3Boto3Storage(S3Boto3Storage, ABC):
    """
    This is our custom version of S3Boto3Storage that fixes a bug in
    boto3 where the passed in file is closed upon upload.
    From:
    https://github.com/matthewwithanm/django-imagekit/issues/391#issuecomment-275367006
    https://github.com/boto/boto3/issues/929
    https://github.com/matthewwithanm/django-imagekit/issues/391
    """

    def _save(self, name, content):
        """
        We create a clone of the content file as when this is passed to
        boto3 it wrongly closes the file upon upload where as the storage
        backend expects it to still be open
        """
        # Seek our content back to the start
        content.seek(0, os.SEEK_SET)

        # Create a temporary file that will write to disk after a specified
        # size
        content_autoclose = SpooledTemporaryFile()

        # Write our original content into our copy that will be closed by boto3
        content_autoclose.write(content.read())

        # Upload the object which will auto close the content_autoclose
        # instance
        super(CustomS3Boto3Storage, self)._save(name, content_autoclose)

        # Cleanup if this is fixed upstream our duplicate should always close
        if not content_autoclose.closed:
            content_autoclose.close()

@pasevin thank you very much for sharing with us this snipped. I'm proposing a little "upgrade" to it to be more pythonic.

class CustomS3Boto3Storage(S3Boto3Storage, ABC):
    """
    This is our custom version of S3Boto3Storage that fixes a bug in
    boto3 where the passed in file is closed upon upload.
    From:
    https://github.com/matthewwithanm/django-imagekit/issues/391#issuecomment-275367006
    https://github.com/boto/boto3/issues/929
    https://github.com/matthewwithanm/django-imagekit/issues/391
    """

    def _save(self, name, content):
        """
        We create a clone of the content file as when this is passed to
        boto3 it wrongly closes the file upon upload where as the storage
        backend expects it to still be open
        """
        # Seek our content back to the start
        content.seek(0, os.SEEK_SET)

        # Create a temporary file that will write to disk after a specified
        # size. This file will be automatically deleted when closed by
        # boto3 or after exiting the `with` statement if the boto3 is fixed
        with SpooledTemporaryFile() as content_autoclose:

            # Write our original content into our copy that will be closed by boto3
            content_autoclose.write(content.read())

            # Upload the object which will auto close the
            # content_autoclose instance
            super(CustomS3Boto3Storage, self)._save(name, content_autoclose)

Thank you @pasevin @vstoykov! Here is the fixed version:

class CustomS3Boto3Storage(S3Boto3Storage, ABC):
    """
    This is our custom version of S3Boto3Storage that fixes a bug in
    boto3 where the passed in file is closed upon upload.
    From:
    https://github.com/matthewwithanm/django-imagekit/issues/391#issuecomment-275367006
    https://github.com/boto/boto3/issues/929
    https://github.com/matthewwithanm/django-imagekit/issues/391
    """

    def _save(self, name, content):
        """
        We create a clone of the content file as when this is passed to
        boto3 it wrongly closes the file upon upload where as the storage
        backend expects it to still be open
        """
        # Seek our content back to the start
        content.seek(0, os.SEEK_SET)

        # Create a temporary file that will write to disk after a specified
        # size. This file will be automatically deleted when closed by
        # boto3 or after exiting the `with` statement if the boto3 is fixed
        with SpooledTemporaryFile() as content_autoclose:

            # Write our original content into our copy that will be closed by boto3
            content_autoclose.write(content.read())

            # Upload the object which will auto close the
            # content_autoclose instance
            return super(CustomS3Boto3Storage, self)._save(name, content_autoclose)

I assume the ABC class inherited by the CustomS3Boto3Storage class is from someone's existing code and shouldn't be in these examples?

Note that although the two classes posted by @vstoykov and @mannpy look identical apart from the syntax highlighting, the one from @vstoykov is missing a return on the final line, which results in it not attaching the uploaded image to your model. This caught me out :) Thanks everyone for finding a solution!

@philgyford thank you for your clarification. I just rewrote the @pasevin variant without testing it in real life.

@mannpy thank you for fixing the snippet to work correctly. I marked my comment as outdated to stay only your variant because people can copy mine by accident and will struggle with errors.

Hey, just chiming in on this issue. While https://github.com/matthewwithanm/django-imagekit/issues/391#issuecomment-592877289 works great, it doesn't work well for large files (e.g. > 1GB) unless your machine has a lot of RAM. In that case you'll want to specifically use this custom storage for ImageKit only via IMAGEKIT_DEFAULT_FILE_STORAGE

To avoid this issue completely, could you defer image generation? Is the root of this problem that the backend storage finishes up before ImageKit is done processing?

@

Getting this issue when creating a new image. I simply have a few specs on an ImageField like this:

    big_image = ImageSpecField(
        [SmartResize(width=900, height=600)],
        source='original_image',
        format='JPEG',
        options={'quality': 75}
    )

Exception:

ValueError` at /api/v1/pictures/pictures/

I/O operation on closed file.
Request Method: POST
Request URL:    https://foobar.com/api/v1/pictures/pictures/
Django Version: 1.10.2
Exception Type: ValueError
Exception Value:    
I/O operation on closed file.
Exception Location: /home/foobar/.virtualenvs/foobar/local/lib/python2.7/site-packages/imagekit/cachefiles/__init__.py in _generate, line 102

Later part of the stack trace, somehow the "File" gets set to none?:

/home/foobar/.virtualenvs/foobar/local/lib/python2.7/site-packages/imagekit/cachefiles/backends.py in generate_now
            file._generate()
Local Vars

Variable    Value
file    
<ImageCacheFile: CACHE/images/pictures/2016/10/25/foobar.jpg>
force   
False
self    
<imagekit.cachefiles.backends.Simple object at 0x7f9e2e91ce50>
/home/foobar/.virtualenvs/foobar/local/lib/python2.7/site-packages/imagekit/cachefiles/__init__.py in _generate
        content.seek(0)
Local Vars

Variable    Value
actual_name 
u'CACHE/images/pictures/2016/10/25/foobar.jpg'
content 
<File: None>
self    
<ImageCacheFile: CACHE/images/pictures/2016/10/25/foobar.jpg>

I have a setup with S3 and this started happening when I updated the boto library to boto3. Reverting back seemed to help. I can provide more info if needed.

Can you tell me how to revert back to boto?? Do I uninstall boto3 and install boto?? Is that all??

Thank you @pasevin @vstoykov! Here is the fixed version:

Can you please tell me where to write this code?? I mean in which file?? and where is this located??

Can you please tell me where to write this code?? I mean in which file?? and where is this located??

You choose. e.g. Make a file at myapp/storages.py containing this (the class is copied from @manpy's comment, with the ABC class removed, and the imports added):

import os
from storages.backends.s3boto3 import S3Boto3Storage
from tempfile import SpooledTemporaryFile

class CustomS3Boto3Storage(S3Boto3Storage):
    """
    This is our custom version of S3Boto3Storage that fixes a bug in
    boto3 where the passed in file is closed upon upload.
    From:
    https://github.com/matthewwithanm/django-imagekit/issues/391#issuecomment-275367006
    https://github.com/boto/boto3/issues/929
    https://github.com/matthewwithanm/django-imagekit/issues/391
    """

    def _save(self, name, content):
        """
        We create a clone of the content file as when this is passed to
        boto3 it wrongly closes the file upon upload where as the storage
        backend expects it to still be open
        """
        # Seek our content back to the start
        content.seek(0, os.SEEK_SET)

        # Create a temporary file that will write to disk after a specified
        # size. This file will be automatically deleted when closed by
        # boto3 or after exiting the `with` statement if the boto3 is fixed
        with SpooledTemporaryFile() as content_autoclose:

            # Write our original content into our copy that will be closed by boto3
            content_autoclose.write(content.read())

            # Upload the object which will auto close the
            # content_autoclose instance
            return super(CustomS3Boto3Storage, self)._save(name, content_autoclose)

Then in your settings.py add something like:

DEFAULT_FILE_STORAGE = "myapp.storages.CustomS3Boto3Storage"

with the path reflecting where you've put the storages.py file in relation to your settings file.

@philgyford & @mannpy This works for me as well! Thanks.

We use Digital Ocean Spaces (S3 compatible) for our media. It seems that the first call to the thumbnail is giving us the error (ie. the uploading), but next calls are fine.

Can you please tell me where to write this code?? I mean in which file?? and where is this located??

You choose. e.g. Make a file at myapp/storages.py containing this (the class is copied from @manpy's comment, with the ABC class removed, and the imports added):

import os
from storages.backends.s3boto3 import S3Boto3Storage
from tempfile import SpooledTemporaryFile

class CustomS3Boto3Storage(S3Boto3Storage):
    """
    This is our custom version of S3Boto3Storage that fixes a bug in
    boto3 where the passed in file is closed upon upload.
    From:
    https://github.com/matthewwithanm/django-imagekit/issues/391#issuecomment-275367006
    https://github.com/boto/boto3/issues/929
    https://github.com/matthewwithanm/django-imagekit/issues/391
    """

    def _save(self, name, content):
        """
        We create a clone of the content file as when this is passed to
        boto3 it wrongly closes the file upon upload where as the storage
        backend expects it to still be open
        """
        # Seek our content back to the start
        content.seek(0, os.SEEK_SET)

        # Create a temporary file that will write to disk after a specified
        # size. This file will be automatically deleted when closed by
        # boto3 or after exiting the `with` statement if the boto3 is fixed
        with SpooledTemporaryFile() as content_autoclose:

            # Write our original content into our copy that will be closed by boto3
            content_autoclose.write(content.read())

            # Upload the object which will auto close the
            # content_autoclose instance
            return super(CustomS3Boto3Storage, self)._save(name, content_autoclose)

Then in your settings.py add something like:

DEFAULT_FILE_STORAGE = "myapp.storages.CustomS3Boto3Storage"

with the path reflecting where you've put the storages.py file in relation to your settings file.

Thanks you very much for sharing this :). It works with: django-imagekit==4.0.2 django-storages==1.11.1 boto3==1.17.12 botocore==1.20.12

@philgyford & @mannpy This works for me as well! Thanks.

We use Digital Ocean Spaces (S3 compatible) for our media. It seems that the first call to the thumbnail is giving us the error (ie. the uploading), but next calls are fine.

Did you resolve this? I have the same setup and it just broke after updating some versions.

I used a custom storageclass like the one @philgyford suggested, which worked fine with versions:

boto3==1.10.2
django-imagekit==4.0.2
django-storages==1.7.2
pilkit==2.0
Pillow==5.4.1
sorl-thumbnail==12.5.0

Currently it does not work for versions:

boto3==1.17.72
django-imagekit==4.0.2
django-storages==1.11.1
pilkit==2.0
Pillow==8.2.0
sorl-thumbnail==12.7.0

@FinnGu, I'm currently using the same code as above successfully with:

boto3==1.17.73
django-imagekit==4.0.2
django-storages==1.11.1
pilkit==2.0
Pillow==8.2.0

I don't have sorl-thumbnail installed. Sorry that this isn't a solution to your problem!

@FinnGu, I'm currently using the same code as above successfully with:

boto3==1.17.73
django-imagekit==4.0.2
django-storages==1.11.1
pilkit==2.0
Pillow==8.2.0

I don't have sorl-thumbnail installed. Sorry that this isn't a solution to your problem!

Well, this is embarrasing... I still used the old version that was proposed here and not the newer version and of course it is working now.

For others that have overlooked this as well: At some point django-storages removed the _save_content() method and the new solution uses _save() instead. Also it is now required to return the super() call in the last line, otherwise nothing gets saved at all.

Finally found the solution here. thank you, guys.

I have the same issue with default FileStorage in development. I serve thumbnail image using django-downloadview==2.1.1 and if thumbnail file does not exist yet and is generated, the view crashes.

I'm using the following solution to re-open the file if it was closed after generation:

class ImagekitCacheStrategy(JustInTime):
    def on_existence_required(self, file: ImageCacheFile):
        super().on_existence_required(file)
        if file.closed:
            file.open()